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Preface 


The  purpose  of  this  study  was  to  determine  if  artificial  networks  could  be  used  to  characterize 
radar  signals  from  sampled  feature  data  of  their  electromagnetic  signals.  Ilyperplane  Classifiers, 
trained  via  backpropagation  to  optimize  the  Mean  Square  Error,  Cross  Entropy  and  Classification 
Figure  of  Merit  objective  functions,  were  first  analyzed  and  tested.  Kernel  Classifiers,  using  radial 
basis  functions  as  the  kernel  functions,  were  then  analyzed  mathematically  and  tested  under  var¬ 
ious  training  algorithms.  Finally,  the  Probability  Neural  Networks  were  analyzed  and  tested  for 
comparison  with  the  Ilyperplane  Classifier  and  Kernel  Classifier  networks. 

The  amount  of  work  accomplished  could  not  have  been  done  wi  ,.out  the  aid  of  my  thesis 
committee,  Dr.  M.  Kabrisky,  Dr.  B.  Sutter,  and  Dr.  V.  Pyatti.  I  am  especially  indebted  to  my 
thesis  advisor  Maj  S.  Rogers  for  his  direction  and  support  during  this  pr‘.>ject.  Also.  T  thank  Capt 
Eddy  and  Anthony  Schooler  for  their  help  with  the  SUN  Workstations  and  LA.'  AX.  Finally,  I 
would  like  to  thank  my  wife  Annette  for  the  sacrifices  she  n*ade  in  order  that  this  mnestone  in  our 
lives  could  be  completed. 


Daniel  R.  Zahirniak 
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Abstract 

Recent  work  concerning- artificial  neural  networks  has  focused  on  decreasing  network  training 
times.  Kernel  Classifier  networks,  using  radial  basis  functions  (RBFs)  as  the  kernel  function,  can 
be  trained  quickly  with  little  performance  degradation.  Short  training  times  are  critical  for  systems 
which  must  adapt  to  changing  environments. 

The  function  of  Kernel  Classifier  hetworks  is  based  on  the  principle  that  multivariate  functions 
can  be  approximated  via  linear  combinations  of  RBFs.  RBFs  can  also  perform  probability  density 
estimations,  making  classifications  approximating  a  Bayers  optimal  descriminant. 


Methods  used  to  set  the  RBF  centers  included  matching  the  training  data,  Kohonen  Training, 
K-Meah'SvClustering  and  placement  at  averages  of  data  clusters  of  the  same  class. 

Test  results  indicate  the  performance  of  these  networks  was  equal  to  that  of  Hyperplane  Clas¬ 
sifier  networks  trained,  via  backpropagation,  to  optimize  the  Mean  Square  Error,  Cross  Entropy, 
and  Classification  Figure  of  Merit  objective  functions.  However,  the  RBF  networks  trained  much 
faster.  The  RBF  networks  also  outperformed  the  Probability  Neural  Netw6rks,(PNN)  indicating 
the  weights  in  the  output  layer  offset  the  choice  of  non-optimal  spreads. 


This  ability  to  train  quickly  while  obtaining  high  classification  accuracies  make  RBF  Ker¬ 
nel  Classifier  networks^an  attractive  option  for  systems  which  must  adapt  quickly  to  changing 
environments. 


Characterization  of  Radar  Signals  Using  Neural  Networks 


I.  Problem  Description 


LI  Iniroduciion 

Due  to  the  increasing  proliferation  of  hostile  radar  systems,  the  current  radar  warning  de¬ 
vices  installed  in  many  Air  Force  aircraft  may  have  trouble  meeting  their  real-time  data  processing 
requirements  in  the  near  future.  Since  artificial  neural  networks  are  designed  to  process  data  in  a 
distributed  manner,  they  will  able  to  process  data  much  quicker  than  current  computer  systems 
when  the  parallel  distributed  processing  hardware  becomes  available  .  Thus,  artificial  neural  net¬ 
works  may  provide  the  key  t'^  solving  the  real-time  data  processing  requirements  of  future  radar 
warning  devices.  This  thesis  will  characterize  several  types  of  artificial  neural  networks  and  de¬ 
termine  if  any  are  suited  to  accurately  characterize  radar  systems.  This  problem  description  will 
begin  by  reviewing  the  background  of  these  radar  warning  devices.  The  exact  problem  to  be  solved 
will  then  be  described.  This  description  will  be  followed  by  a  summary  of  the  knowledge  cur¬ 
rently  available  concerning  artificial  neural  networks  and  a  brief  outline  of  the  scope  of  the  thesis. 
The  standards  and  approach  taken  to  solve  the  problem  will  then  be  discussed.  This  chapter  will 
conclude  with  a  brief  overview  of  the  remaining  chapters. 

L2  Background 

The  main  task  required  of  a  radar  warning  system  is  to  analyze  the  electromagnetic  environ¬ 
ment,  determine  if  this  environment  contains  a  radar  signal,  classify  the  signal  as  being  generated 
from  either  a  hostile  or  friendly  radar  system,  and  identify  the  exact  type  of  radar  system  trans¬ 
mitting  this  signal  or  classify  the  signal  as  being  from  an  unknown  emitter.  This  analysis  should  be 
accomplished  even  though  the  radar  system  has  the  ability  to  change  the  signals  electromagnetic 
characteristics  (19:42).  Accomplishing  this  task  requires  the  radar  warning  system  to  analyze  the 
environment  quickly  and  accurately,  classifying  signal  data  according  to  known  parameters.  Even 
with  the  use  of  high  speed  computers,  this  task  is  computationally  intense  and  can  take  several 
seconds  to  properly  classify  the  radar  sigiial.  As  the  electromagnetic  spectrum  becomes  increasingly 
crowded  through  the  proliferation  of  hostile  radar  systems,  it  will  take  longer  for  the  current  radar 
warning  devices  to  accomplish  their  mission.  However,  aircraft  are  required  to  operate  in  real-time, 
responding  to  changes  in  this  electromagnetic  environment  in  a  matter  of  milliseconds.  This  has 


led  to  research  in  the  use  of  artificial  neural  networks  as  a  possible  method  of  characterizing  a  radar 
system  from  its  transmitted  signal. 

LS  Problem 

This  thesis  will  characterize  several  different  types  of  artificial  neural  networks  and  determine 
which  would  be  able  to  classify  radar  systems  from  data  concerning  their  electromagnetic  signals. 
An  artificial  neural  network  can  be  thought  of  as  a  massively  parallel,  interconnected  system  of 
simple  computing  elements,  called  nodes,  which  can  accomplish  certain  pattern  classification  tasks 
quickly,  in  a  way  motivated  by  a  biological  nervous  system.  Since  neural  networks  have  been  shown 
to  be  equivalant  to  a  Bayes^  optimal  discriminant  (21)  and  capable  of  performing  arbitrary  complex 
transformations  (5),  it  seems  logical  to  assume  these  networks  will  be  able  to  characterize  radar 
systems. 

1.4  Current  Knowlettg'e 

/  Basically,  artificial  neural  networks  may  be  categorized  as  either  Probabilistic  Classifiers, 
Exemplar  Classifiers,  Hyperplane  Classifiers,  or  Kernel  Classifiers  (11:47-63).  A  Probabilistic  Clas¬ 
sifier  neural  network  seeks  to  classify  patterns  by  using  probability  distributions  to  maximize  the 
probabilities  associated  with  a  classification.  As  such,  these  networks  require  an  assumption  of  the 
probability  distribution  of  the  input  data  (9:1-7).  An  Exemplar  Classifier  neural  network  classifies 
unknown  feature  data  based  on  a  nearest-neighbor  calculation  with  the  training  data.  That  is,  the 
closer  an  unknown  data  point  is  to  a  known  data  point  in  the  feature  space,  the  stronger  the  prob¬ 
ability  that  the  two  features  represent  the  same  object  (8:167-170).  A  Hyperplane  Classifier  neural 
network  fornfis  decision  regions  by  using  hyperplaues  to  partition  the  feature  space  into  regions  of 
interest.  This  partitioning  allows  the  network  to  make  classifications  of  similar  data  (19:48-63). 
A  Kernel  Classifier  neural  network  uses  overlapping  kernel  function  nodes  to  create  complex  deci¬ 
sion  regions  over  the  feature  space.  These  decision  regions  will  determine  the  classification  of  each 
pattern  as  similar  patterns  will  be  identified  within  the  decision  regions  (11:49). 

Usually,  these  artificial  neural  networks  are  developed  using  unsupervised  training,  supervised 
training,  or  a  combination  of  supervised  and  unsupervised  training.  In  unsupervised  training,  the 
feature  data  from  the  environment  arc  input  to  the  network.  The  nodes  in  the  ncUvork  arc  then 
allowed  to  arrange  their  parameters,  or  cluster,  in  positions  reflecting  the  distribution  of  the  data 
(22:151-193).  In  supervised  training,  the  feature  data,  in  the  form  of  a  pattern  vector,  is  presented 
to  the  network  along  with  the  desired  output  pattern  for  that  particular  input  pattern  vector.  The 
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difference,  or  error,  between  the  network  output  and  the  desired  output  is  then  calculated  and  used 
to  adjust  the  network  parameters  in  such  a  way  that  the  error  is  minimized.  A  combination  of 
unsupervised  and  supervised  training  can  also  be  used  to  develop  an  artificial  neural  network.  In 
this  type  of  training,  the  network  is  first  trained  using  unsupervised  training  to  allow  the  network 
parameters  to  be  distributed  according  to  the  feature  data.  After  stabilization,  the  network  is  then 
trained,  in  a  supervised  fashion,  to  produce  the  correct  classification  for  a  given  input  pattern. 

1.5  Scope 

The  final  product  of  this  thesis  will  be  a  characterization  of  the  Hyperplane  and  Kernel 
Classifier  neural  networks  and  a  determination  if  they  can  accurately  characterize  radar  signals. 
The  Probabilistic  Classifier  will  be  briefly  studied  for  comparison  purposes.  The  Exemplar  Classifier 
will  not  be  studied  as  the  computational  time  required  to  make  a  classification  would  exceed  the 
real-time  data  processing  requirements  of  the  aircraft.  The  Hyperplane  Classifier  networks  will 
consist  of  a  double  hidden  layer  network,  as  shown  in  figure  1.1.  This  network  will  be  trained, 


via  backpropagation,  to  optimize  the  Mean  Squared  Error,  (MSE),  the  Cross  Entropy  (CE)  and 
the  Classification  Figure  of  Merit  (CFM)  objective  functions  (28).  The  transfer  function  for  each 
network  node  will  be  sigmoidal.  The  Kernel  Classifier  will  be  a  single  hidden  layer  network,  as 
shown  in  figure  1.2,  developed  using  the  combined  training  method  with  radial  basis  functions  as 
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kernel  functions.  The  transfer  function  for  each  of  the  kernel  nodes  in  the  hidden  layer  will  be 
gaussian.  The  transfer  function  for  the  nodes  in  the  output  layer  will  be  linear. 

Several  methods  will  be  used  to  determine  the  weights  linking  the  network's  hidden  layer 
nodes  to  the  input  layer  nodes.  The  first  method  will  set  the  weights  at  values  equal  to  the  features 
of  the  training  set  patterns.  The  second  method  will  set  the  weights  via  the  Kohonen  training 
algorithm.  The  third  method  will  set  the  weights  via  a  K-means  clustering  algorithm.  The  fourth 
method  will  set  the  weights  at  the  average  of  clusters  within  the  pattern  classes.  The  weights 
linking  the  kernel  nodes  in  the  hidden  layer  with  the  nodes  in  the  output  layer  will  be  established 
by  a  global  minimization  of  the  MSE  function. 

J.6  Standards 

The  performance  criteria  for  each  network  will  be  the  classification  accuracy  and  the  amount 
of  tirrie  it  takes  to  train  the  network.  Classification  accuracy  is  the  more  important  of  the  two 
performance  criteria  since  near  perfect  classification  accuracy  is  a  mission  requirement.  The  amount 
of  time  it  takes  to  train  the  network  is  important  since  it  is  highly  likely  that  the  threats  in  the 
environment  can  change  from  mission  to  mission.  Short  training  times  can  be  crucial  to  building 
networks  to  adapt  to  this  changing  environment.  The  accuracy  of  each  trained  network  will  be 
calculated  by  applying  test  data  to  the  network,  and  allowing  the  network  to  make  a  classification. 
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An  error  will  result  when  the  network’s  classification  does  not  match  the  known  classification.  The 
accuracy  will  then  be  calculated  as  the  ratio  of  the  number  of  correct  classifications  to  the  number 
of  input  patterns. 

J,7  Approach 

As  part  of  this  thesis,  a  software  environment  will  be  developed  on  the  SUN  graphic  work¬ 
stations.  This  software  will  be  written  in  ANSI  C  and  designed  according  to  an  object-oriented 
approach.  The  software  will  allow  the  user  to  select  the  number  of  layers  for  the  neural  network 
and  to  select  the  training  rule  for  each  layer.  In  this  manner,  different  combinations  of  training 
rules  can  be  combined  and  their  overall  performance  evaluated.  Furthermore,  this  software  will 
allow  the  user  to  select  either  the  sigmoidal,  gaussian  or  linear  transfer  functions  for  each  node  in 
the  network.  This  will  allow  construction  of  many  different  types  of  networks  even  though  their 
topology  may  be  the  same.  Since  the  main  task  of  this  thesis  is  in  pattern  recognition,  this  software 
will  implement  only  strict  feed  forward  networks.  However,  the  software  will  be  designed  to  allow 
future  expansion  to  recurrent  and  lateral  inhibition  networks. 


L8  Chapter  Outlines 

The  following  is  a  brief  discussion  of  the  information^.,0  be  found  in  each  of  the  chapters  of 
the  thesis. 

L8J  Chapter  2  This  chapter  will  provide  a  review  pattern  recognition  in  general  followed 
by  a  brief  synopsis  of  biological  and  artificial  neural  networks.  This  chapter  will  then  conclude  with 
a  history  of  the  development  of  artificial  neural  networks,  a  description  of  the  training  methods 
used  to  implement  these  networks  and  a  discussion  of  the  different  classifications  of  artificial  neural 
networks. 

1,8,2  Chapter  S  This  chapter  will  provide  a  mathematical  analysis  of  the  algorithms  to  be 
used  to  train  the  neural  networks  implemented  as  part  of  this  thesis.  In  particular  the  backprop- 
agation  learning  algorithms  for  the  MSB,  GE  and  CFM  objective  functions,  will  be  derived  for  the 
perceptron-based  Hyperplane  Classifier  networks.  The  algorithms  developed  for  the  radial  basis 
function  Kernel  Classifier  networks  will  concentrate  on  first  setting  the  weights  of  the  hidden  layer 
nodes  prior  to  establishing  the  weights  in  the  output  layer  nodes  via  a  global  minimization  of  the 
MSB  objective  function. 


1-5 


1.8.3  Chapter  4  This  chapter  will  provide  a  detaik  ^  analysis  of  the  software  developed  for 
this  thesis.  This  will  include  a  discussion  of  the  software  structure  and  a  mapping  ofthe  algorithms 
developed  in  Chapter  3. 

1.8.4  Chapter  5  This  chapter  will  discuss  the  testing  and  results  for  each  of  the  networks 
implemented  with  the  software  described  in  Chapter  4.  This  will  include  an  overview  of  the  classi¬ 
fication  problem,  a  discussion  of  the  data  used  to  train  and  test  the  network  and  an  analysis  of  the 
results. 


1.8,5  Chapter  6  This  chapter  will  provide  conclusions  based  on  the  results  detailed  in  Chap¬ 
ter  5  and  include  recommendations  for  areas  of  future  study. 

1,9  Summary 

This  thesis  will  characterize  the  performance  of  several  artificial  neural  networks  and  deter¬ 
mine  if  the  networks  can  be  trained  to  accurately  classify  radar  systems  from  data  concerning  their 
electromagnetic  signals.  For  this  thesis,  Hyperplane  Classifier,  Kernel  Classifier,  and  Probabilistic 
Classifier  networks  will  be  developed  and  tested  to  analyze  their  performance.  The  results  of  this 
thesis  will  provide  a  determination  of  the  feasibility  of  using  artificial  neural  networks  as  the  basis 
for  the  Air  Forceps  next  generation  of  radar  warning  devices. 
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IL  Literature  Review 


2J  Iniroduciion 

Current  computer  systems  may  not  be  able  to  process  data  fast  enough  to  meet  the  future 
real-time  performance  requirements  of  many  military  weapons  systems.  Since  artificial  neural 
networks  are  designed  to  process  data  in  a  distributed  manner,  they  may  provide  the  key  to  solving 
these  data  processing  requirements.  This  literature  review  begins  with  the  background  associated 
with  the  military’s  strict  data  processing  requirements  followed  by  a  brief  review  of  the  concepts  of 
pattern  recognition.  Research  showing  how  the  brain  may  use  biological  neural  networks  to  process 
information  is  then  examined.  After  describing  artificial  neural  networks  in  general,  this  review  will 
cover  some  of  the  important  milestones  in  the  development  of  artificial  neural  networks.  Finally, 
this  literature  review  will  describe  the  current  methods  used  to  train  these  networks  and  discuss 
how  artificial  neural  networks  are  now  being  categorized  according  to  their  methods  of  classifying 
data. 


2.2  Background 

A  task  required  of  many  military  weapon  systems  is  to  analyze  the  environment  and  deter¬ 
mine,  in  a  matter  of  milliseconds,  whether  a  target  of  interest  is  present.  Solving  this  problem  of 
pattern  recognition  usually  requires  the  weapon  system  to  combine  the  data  from  a  multitude  of 
sensors,  segment  the  data  into  areas  of  interest,  extract  the  important  features,  and  classify  these 
features  according  to  known  threat  patterns  (20:1-12).  Currently,  these  pattern  recognition  tasks 
are  accomplished  using  the  traditional  Von  Neumann  computers.  These  computers  consist  of  a 
Central  Processing  Unit  which  performs  complex  sequential  computations,  one  at  a  time,  under 
control  of  a  system  clock  (6:2).  Even  with  the  use  of  high  speed  computers,  this  task  of  pattern 
recognition  is  usually  so  computationally  intense  that  it  may  take  hours  to  properly  classify  the 
pattern  (26:7).  These  time  frames  are  unacceptable  for  weapon  systems  operating  in  real  time. 
The  only  object  capable  of  performing  this  type  of  analysis  in  real-time  is  the  human  brain.  Thus, 
a  computing  architecture,  based  upon  the  way  the  brain  is  assumed  to  function,  may  be  able  to 
solve  these  pattern  recognition  problems  in  the  time  frames  required  (26:7).  These  computing 
architectures,  commonly  known  as  artificial  neural  networks,  perform  computations  in  a  manner 
significantly  different  than-traditional  computers.  For4nstance,  an  artificial  neural  network^proce.ss- 
ing  unit  may  do  only  one  type  of  simple  calculation,  such  as  producing  a  single  output  from  a  simple 
transformation  of  its  inputs.  However,  since  there  may  be  thousands  of  these  simple  processing 
units,  each  interconnected  to  many  others,  extremely  complex  computations  can  be  accomplished. 
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in  parallel,  via  the  network  as  a  whole.  This  parallel  processing  of  data  allows  the  artificial  neural 
network  to  process  data  extremely  quickly  and  may  provide  the  key  to  rapidly  classifying  patterns 
from  their  distinct  environmental  features. 

2,S  Paiitrn  Recogniiion 

Recognition  of  patterns  is  a  basic  characteristic  of  many  living  organisms,  including  human 
beings  (27:5).  The  fact  that  a  human  being  is  a  very  sophisticated  information  processing  system  is 
primarily  due  to  the  fact  that  human  beings  possess  a  superior  pattern  recognition  capability  (27:5). 
That  is,  even  though  our  senses  are  constantly  flooded  with  an  overwhelming  variety  of  patterns 
from  the  environment,  we  still  have  the  ability  to  determine  what  information  is  most  important  and 
react  accordingly.  It  is  this  characteristic  of  discriminating  unknown  patterns  between  populations 
that  a  pattern  recognition  system  seeks  to  emulate. 

2,4  Paitem  Recogniiion  Systems 

Basically,  all  pattern  recognition  systems  seek  to  categorize  the  input  data  into,  identifiable 
classes  via  the  extraction  of  significant  features  from  a  background  of  irrelevant  detail  (27:6).  Thus, 
the  tasks  of  a  pattern  recognition  system  are  to  sense  the  environment,  provide  data  concerning 
patterns  of  possible  interest,  extract  relevant  features  from  this  data,  and  classify  the  pattern  as  a 
member  of  one  of  the  groups  under  consideration.  These  tasks  are  shown  in  the  block  diagram  of 
figure  2.1. 

2,4»J  Sensing  The  first  task  of  a  pattern  recognition  system  is  to  represent  the  pattern 
under  study  as  a  group,  or  vector,  of  measurements.  This  process,  known  as  sensing,  attempts 
to  describe  the  characteristics  of  the  pattern  under  study  and  represent  the  pertinent  information 
available  about  the  pattern  (27:9).  For  example,  if  the  task  was  to  recognize,  or  classify  a  radar 
signal,  the  measurements  of  the  pulse  repetition  interval,  scan  rate  and  operating  frequency  might 
be  taken. 

2,4^^  Feature  Selection  The  second  task  of  a  pattern  recognition  system  is  to  take  the 
measured  data  obtained  under  sensing  and  extract  the  intraset  and  interset  features,  or  attributes, 
which' will  enable  the  system  to  perform  classification.  This  step  is  perhaps  the  most  critical  as 
good  features  make  for  good  classification  systems  (19:47).  However,  at  this  time,  there  is  no  strict 
set  of  rules  available  to  determine  which  features  actually  characterize  a  class.  Too  few  or  poorly 


2-2 


Figure  2.1.  Block  Diagram  of  a  Pattern  Recognition  System  (27:16) 


chosen  features  will  not  allow  the  system  to  characterize  the  input  patterns  sufficiently  to  allow 
categorization  (27:7). 

2,4 Caiegorizaiion  The  third  task  of  a  pattern  recognition  system  is  to  classify,  or  cate¬ 
gorize,  the  input  pattern  as  belonging  to  one  of  a  set  of  possible  classes.  In  this  step,  the  features 
extracted  from  the  unknown  pattern  are  analyzed  and  used  to  decide  from  which  class  the  unknown 
pattern  is  mostly  likely  to  be  a  member.  This  categorization  is  usually  based  upon  some  decision 
function  such  as  Bayes’  optimal  discriminant. 

Currently,  most  of  these  pattern  recognition  tasks  are  accomplished  via  a  variety  of  classical 
statistical  methods  such  as  template  matching,  frequency  histogram  associations,  and  probability 
density  estimations.  These  methods  can  require  much  contextual  analysis  of  the  data  prior  to 
classification.  However,  artificial  neural  networks  can  be  constructed  as  pattern  recognition  systems, 
adapting  their  internal  parameters  according  to  a  set  or  predefined  rules,  without  the  need  for 
intensive  human  analysis  of  the  data.  To  understand  how  this  process  may  occur  requires  an 
understanding  of  biological  neural  networks. 
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2.5  Biological  Neural  NeUvorks 

Biologically  speaking,  a  neuron,  as  shown  in  figure  2.2,  is  a  nerve  cell  which  is  used  to  process, 
store,  retrieve  and  manipulate  information  received  from  the  environment  (26:9).  This  neuron  is 


a  cell  that  has  been  modified  to  become  a  simple  processing  element  whose  primary  function  is 
to  receive,  process  and  transmit  electrochemical  signals  across  the  brain^s  neural  pathways  (29:12) 
(19:17).  The  main  modifications  are  the  addition  of  dendrite  and  axon  appendages  to  the  cell 
body.  The  dendrites  act  as  the  input  communication  channels  while  the  axon  acts  as  the  output 
channel.  Each  neuron  is  connected  to  the  axon  of  many  other  neurons  via  its  dendrites.  These 
dendrite  extensions  allow  a  neuron  to  receive  chemical  neurotransmissions  from  other  neurons  at 
junctions  called  synapses  (29:12).  When  the  neuron  receives  these  signals,  it  will  become  excited  if 
the  combined  input  signals  exceed  a  threshold.  When  excited,  the  neuron  will  transmit  an  electrical 
signal  along  its  axon,  sending  the  signal  to  each  attached  neuron.  The  attached  neurons  may  or 
may  not  become  excited,  depending  on  the  strength  of  the  connection  between  the  neurons.  Thus, 
it  is  in  the  synapse  that  the  information  is  stored  in  the  form  of  synaptic  weights.  Since  there  are 
between  10^®  -  10^^  neurons  in  the  human  brain  and  an  estimated  10^®  interconnections  between 
these  neurons,  a  vast  amount  of  information  can  be  stored  and  quickly  processed  (29:12). 
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2*6  Artificial  Neural  Networks 


As  a  pattern  recognition  device,  artificial  neural  networks  attempt  to  assign  an  unknown 
pattern  from  the  environment  into  one  of  a  set  of  selected  classes  by  emulating  this  structure  of 
the  neurons  in  the  human  brain  (11:47).  These  networks  can  be  thought  of  as  massively  parallel, 
interconnected  networks  of  simple  computing  elements,  called  nodes,  which  seek  to  interact  with 
the  real  world  in  way  similar  to  a  biological  nervous  system  (10:251).  As  shown  in  figure  2.3,  each 
node  in  the  network  performs  a  simple  transformation  of  inputs  from  other  nodes  in  the  network, 
or  from  the  environment,  to  produce  a  single  output  signal.  This  transformation  can  be  via  a 


Figure  2.3.  Artificial  Neural  Network  Node  (19:48) 


linear  function  or  a  nonlinear  function  such  as  sigmoidal,  gaussian,  or  threshold  function.  The 
signal  output  from  this  transformation  is  then  fed  to  other  nodes  or  interpreted  as  the  output  of 
the  network. 

The  connections,  or  weights  between  the  nodes,  function  in  a  manner  similar  to  the  axon- 
dendrite  synaptic  connection  of  biological  neurons.  That  is,  each  weight  has  a ’^strength”  associated 
with  it  which  serves  to  either  amplify  or  inhibit  the  signals  transmitted  between  nodes  along  these 
connections.  Typically,  an  artificial  neural  network  will  consist  of  one  or  more  layers  of  nodes  as 
shown  in  figure  2.4.  The  first  layer  of  nodes  serve  to  simply  pass  the  input  features,  via  weighted 
interconnections,  to  feature  detector  nodes  in  the  second  layer.  These  feature  detector  nodes  will 
usually  respond  to  certain  features  of  the  input  data.  Their  responses  are  then  passed,  via  another 
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Figure  2.4.  Multilayer  Artificial  Neural  Network  (6:75) 


set  of  weighted  interconnections,  to  the  output  layer  which  performs  the  classification  task  based  on 
the  outputs  of  the  feature  detectors.  Through  the  proper  interconnection  of  nodes  and  weights,  and 
through  the  use  of  an  appropriate  transfer  function,  artificial  neural  networks  have  been  developed 
which  can  accurately  classify  patterns  ranging  from  phonemes  to  tanks  (23:461-466)  (20:1-7). 

2, 7  Historical  Review 

Biological  neural  networks  have  been  studied  for  years.  Ever  since  the  18th  century,  when 
Galvani  investigated  the  connection  between  electricity  and  the  frog’s  central  nervous  system,  man 
has  been  seeking  to  unlock  the  secrets  of  the  brain’s  computing  power  (19:5).  From  Santiago  Cajal’s 
discovery  of  the  dense  interconnection  of  neurons  in  the  cortex  to  the  first  estimation  of  a  neuron’s 
transfer  function  via  experiments  with  the  Limulus’  photoreceptors,  man  has  been  seeking  a  method 
of  modeling  the  function  of  the  brain  in  the  form  of  artificial  neural  networks  (19:6-31). 

The  first  major  milestone  in  the  development  of  artificial  neural  networks  came  in  1943 
from  McCulloch  and  Pitts.  They  showed  how  neural-like  networks,  using  a  simple  two  state  logical 
decision  element  which  modeled  the  first  order  characteristics  of  a  neuron,  could  compute  a  Boolean 
function  (19:9).  Since  Turing  later  showed  that  any  computable  function  could  be  computed  with 
Boolean  Logic,  the  basis  for  the  development  of  computing  machines,  based  on  the  principle  of 
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using  a  dense  connection  of  simple  neural  like  elements,  was  established.  However,  McCulloch  and 
Pitts  did  not  show  how  a  network.made  of  these  elements  could  be  made  to  ”learn”  (22:152). 

This  problem  of  learning  was  addressed  by  Donald  Hebb  in  1949.  Basically,  Hebb  proposed 
that  the  strength  or  weight  between  two  neurons  be  increased  whenever  both  the  presynaptic  and 
postsynaptic  units  were  active  simultaneously.  These  ideas  remained  untested  due  to  the  lack  of 
technology  capable  of  implementing  these  theories.  (22:152-153). 

M.  Minsky  and  D.  Edmonds  were  the  first  to  actively  implement  Hebb^s  ideas  in  the  form 
of  a  learning/computational  machine  developed  in  1951.  This  machine  was  composed  of  tubes, 
motors  and  electrical  clutches.  The  machine's  memory  was  stored  in  the  positions  of  control  knobs 
by  which  the  machine  adjusted  itself  (22:153). 

This  milestone  was  followed  by  Rosenblatt’s  introduction  of  the  perceptron  in  1957.  Basically, 
the  perceptron  is  a  single  unit  which  produces  an  output  only  when  the  weighted  sum  of  its  inputs 
exceeds  some  preset  threshold.  The  function  of  the  perceptron  was  modeled  on  the  first  order 
characteristics  of  neurons.  Using  Hebb’s  ideas  as  a  basis  for  developing  his  learning  algorithms, 
Rosenblatt  proved  that  the  perceptron  could  learn  anything  it  could  represent  (29:29).  Rosenblatt 
also  helped  pioneer  the  simulation  of  the  perceptron  using  digital  computers  and  developed  a 
set  of  rules  that  would  allow  the  perceptron  to  learn  (19:13).  In  reference  to  pattern  recognition, 
Rosenblatt  showed  that  a  two  layer  perceptron  could  carry  out  any  of  the  2^^  possible  classifications 
of  N  binary  inputs  using  2^  perceptrons  (22:158). 

In  1959,  Bernard  Widrow  invented  the  adaptive  linear  neuron  (ADALINE)  which,  in  a  manner 
similar  to  Rosenblatt’s  perceptron,  would  output  a  signal  only  when  the  weighted  sum  of  its  inputs 
exceeded  a  preset  threshold.  The  weight  parameters  of  the  ADALINE  were  adjusted  over  time  using 
equations  based  on  Hebb’s  original  ideas  (6:31).  Widrow  implemented  ADALINE  based  systems 
that  could  predict  the  weather  and  balance  a  broom  on  a  moving  platform  (19:12). 

These  developments  led  to  an  increased  level  of  activity  in  the  field  of  artificial  neural  networks 
until  1969.  It  was  in  this  year  that  Minsky  and  Papert  proved  that  the  single  layer  perceptron  could 
not  classify  patterns  wJth  features  in  disjoint  regions  in  the  feature  space.  An  example  of  such  a 
pattern  is  shown  in  figure  2.5.  At  this  time,  no  method  of  updating  the  weights  for  any  nodes  except 
those  weights  attached  to  the  output  layer  nodes  for  a  multilayer  perceptron  had  been  established 
(6:29).  Thus,  all  perceptron  based  networks  at  this  time  were  limitcd  to^a  single  input  layer  and 
output  layer  with  only  one  set  of  adjustable  weights.  Minsky’s  and  Pappert’s  proof  helped  stifle 
neural  network  research  until  the  middle  1970’s. 
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Interest  in  artificial  neural  networks  was  rekindled  in  1974.  It  was  at  this  time  that  Paul 
Werbos  introduced  a  method  of  updating  the  weights  in  the  hidden  layers  of  a  multilayer  perceptron 
network.  These  equations  allow  the  multilayer  perceptron  to  overcome  the  disjoint  region  problem 
suffered  by  the  single  layer  perceptron  network. 

However,  little  work  continued  to  be  done  in  the  field  until  1982  when  John  Hopfield  devel¬ 
oped  an  artificial  neural  network  capable  of  providing  associative  memory  and  solving  optimization 
problems  (6:37).  This  development  led  to  increased  activity  in  the  field  of  artificial  neural  networks 
and  the  development  of  many  different  types  of  training  rules  and  architectures. 

2.8  Network  Training 

Artificial  neural  networks  are  not  prograrhmed,  as  are  traditional  computers,  and  there  is 
little  need  for  the  development  of  application  specific  algorithms  to  perform  the  classification  task. 
Artificial  neural  networks  learn  by  example  (6:11).  In  pattern  recognition  tasks,  artificial  neural 
networks  are  usually  trained  to  produce  a  desired  output  whenever  a  known  input  is  applied.  This 
training  is  accomplished  by  applying  input  patterns  to  the  network  and  allowing  the  network  nodes 
to  adjust  their  parameters  in  a  predetermined  fashion  (29:22).  That  is,  the  artificial  neural  network 
is  presented  with  data  which  characterize  such  patterns  as  images,  speech  signals  and  radar  signals. 
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These  networks  are  then  allowed  to  adjust  their  internal  parameters,  such  as  weights,  to  allow  the 
network  to  discover  the  distinguishing  features  heeded  to  perform  a  classification  task  (6:13). 

Basically,  there  are  three  primary  methods  of  developing  or  training  artificial  neural  net¬ 
works.  These  methods  consist  of  unsupervised  training,  supervised  training,  or  combined  training, 
a  combination  of  supervised  and  unsupervised  training. 

2»8J  Unsupervised  Training  In  unsupervised  training,  the  feature  data  from  the  environ¬ 
ment  are  fed  to  the  network.  The  interconnection  weights  between  the  nodes  in  the  network  are 
then  arranged,  or  clustered,  into  positions  reflecting  the  distribution  of  the  training  data  (22:151- 
193).  After  training  is  completed,  application  of  an  input  from  a  given  class  will  produce  a  specific 
output.  However,  there  is  no  way,  before  training,  to  predetermine  the  mapping  from  input  to  out¬ 
put.  A  Kohonen  Self-Organizing  feature  map  network,  trained  in  this  manner,  has  proven  feasible 
for  classifying  speech  patterns  (1:1-7). 

2J.2  Supervised  Training  In  supervised  training,  the  feature  data  is  presented  to  the  net¬ 
work,  along  with  the  desired  output  pattern  for  that  particular  input  pattern.  The  difference 
between  the  network  output  and  the  desired  output,  or  error,  is  then  calculated  and  used  to  adjust 
the  network  parameters,  such  as  the  weights  linking  the  nodes,  so  that  this  error  is  minimized.  This 
process  is  repeated  continuously  until  the  network  is  able  to  produce  the  mapping  from  the  input 
pattern  to  the  desired  output  pattern.  A  multi-layer  perceptroh  network,  developed  with  super¬ 
vised  training  at  the  Air  Force  Institute  of  Technology,  has  proven  capable  of  classifying  tactical 
targets  such  as  trucks,  tanks,  and  jeeps  (20). 

2.8.3  Combined  Training  A  combination  of  unsupervised  and  supervised  training  can  also 
be  used  to  develop  an  artificial  neural  network.  In  this  type  of  network,  the  first  layer  is  usually 
trained  through  unsupervised  training,  allowing  the  network  parameters  to  be  distributed  according 
to  the  feature  data.  After  stabilization  of  the  first  layer,  the  remaining  nodes  are  then  trained  in  a 
supervised  fashion,  to  produce  a  desired  signal  from  knowledge  of  the  distribution  of  the  network 
parameters  in  the  first  layer.  A  network  trained  in  this  manner  has  been  studied  by  the  Royal 
Naval  Engineering  College  as  a  possible  method  of  classifying  radar  signals  (2:1-4). 


2.9  Network  Categorization 

According  to  Lippmann,  neural  networks  may  be  categorized  as  either  a  Probabilistic  Clas¬ 
sifier,  an  Exemplar  Classifier,  a  Hyperplane  Classifier,  or  a  Kernel  Classifier,  depending  upon  the 
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method  the  network  uses  to  perform  classification  (11:47-63). 


2,9 J  Probabilisiic  Classifiers  A  Probabilistic  Classifier  neural  network  seeks  to  classify  pat¬ 
terns  by  using  probability  distributions  to  maximize  the  probabilities  associated  with  a  classification 
as  shown  in  figure  2.6.  As  such,  these  networks  require  enough  training  data  to  allow  an  assump¬ 


tion  of  the  probability  distributions  of  the  patterns  to  be  made.  Either  unsupervised  or  supervised 
training  is  then  used  to  train  the  network.  These  networks  perform  best  when  the  assumed  dis¬ 
tributions  are  accurate  models  of  the  test  data.  An  example  of  this  type  of  classifier  is  the  Bayes* 
Classifier  or  the  Probabilistic  Neural  Network  (9:1-7)  (24). 

2,9,2  Exemplar  Classifiers  An  Exemplar  Classifier  neural  network  classifies  unknown  fea¬ 
ture  data  based  on  a  nearest-neighbor  calculation  with  the  training  data  fixed  in  the  feature  space 
as  shown  in  figure  2.7.  These  nearest-neighbor  calculations  allow  an  estimation  of  the  conditional 
probability  density  functions  for  each  class  (8:166-169).  That  is,  the  closei  unknown  pattern  is 
to  a  known  pattern,  the  stronger  the  probability  that  the  two  patterns  represent  the  same  class 
(10:3).  An  example  of  this  type  of  network  is  the  K-Nearest-Neighbor  classifier.  Networks  of  this 
type  can  be  trained  rather  quickly  through  either  supervised  or  unsupervised  methods,  but  can 
require  large  amounts  of  memory  and  long  computational  times  for  classification  (11:49). 
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2,9.2  Hyperplane  Classifiers  A  Hyperplane  Classifier  neural  network  forms  decision  regions 
by  using  hyperplanes  partition  the  feature  space  into  the  regions  of  interest  as  shown  in  figure 

2.8.  Perhaps  one  of  the  more  studied  networks  of  this  type  is  the  multilayer  perceptron  network 
(19:44-63).  The  function  of  this  network  is  based  on  the  property  that  any  multivariate  function 
can  be  approximated  by  a  finite  superposition  of  sigmoidal  functions  (5:303-313).  This  property 
can  be  implemented  with  21  single  hidden  layer  neural  network,  where  each  node  within  the  network 
uses  a  sigmoidal  function  to  calculate  its  output  from  the  sum  of  the  product  of  its  inputs  and  their 
associated  weights.  This  network  is  usually  trained  under  the  supervised  training  method,  using  a 
technique  called  backp^opagation,  to  minimize  the  error  between  a  given  input  and  a  desired  output 
(19:104-114).  However,  the  amount  of  time  required  to  train  these  networks  can  take  hours  (23:466) 
(14:4).  Once  trained,  Ihese  networks  usually  provide  high  accuracy  for  pattern  classification  while 
requiring  relatively  short  computational  times  (11:49). 

2,9,4  Kernel  Classifiers  A  Kernel  Classifier  neural  network  uses  overlapping  kernel  func¬ 
tions  to  create  complex  receptive-field  decision  regions  over  the  feature  space  as  shown  in  figure 

2.9.  One  of  the  more  recent  types  of  Kernel  function  classifiers  is  the  radial  basis  function  clas¬ 
sifier.  The  function  of  this  network  is  based  on  the  property  that  any  multivariate  function  can 
be  reasonably  approximated  using  a  linear  combination  of  radial  basis  functions  centered  on  the 
data  points,  or  a  subset  of  the  data  points  (17:143-167)  (15:1-20)  (16:978-980).  This  translates 
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Figure  2.9.  Kernel  Classifier  Network  Decision  Regions  (11:49) 
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into  the  establishment  of  a  single  hidden  layer  network,  with  the  nodes  in  the  hidden  layer  using 
radial  basis  functions  to  transform  their  inputs  to  outputs.  These  networks  have  been  successfully 
trained  to  classify  phoneme  data  for  speech  processing  (23:461-466)  (18:437-439)  (14:1-14).  The 
most  appealing  characteristic  of  the  radial  basis  function  Kernel  Classifier  networks  is  the  almost 
instantaneous  training  times  involved  with  setting  the  network  parameters  (23:461-466)  (18:432- 
439).  These  networks  can  also  be  made  to  adapt  to  new  data  by  adding  additional  nodes  as  required 
(12:3).  Generally,  Kernel  Classifiers  can  be  trained  relatively  quickly  through  either  supervised  or 
unsupervised  methods  and  have  intermediate  memory  and  computational  requirements  (11:49). 

2J0  Summary 

Artificial  neural  networks,  may  provide  military  weapon  systems  with  the  ability  to  accurately 
characterize  their  operating  environment  in  real  time.  These  networks  seek  to  emulate  the  function 
of  the  brain;  using  a  dense  connection  of  simple  computational  elements  called  nodes  to  perform 
pattern  classification.  These  networks  can  be  trained  in  either  a  supervised  or  unsupervised  fashion, 
or  both,  and  can  be  categorized  by  the  method  used  to  classify  the  data. 
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IIL  Mathematical  Analysis 


3.1  Introduction 

The  main  function  of  a  pattern  recognition  system  is  to  make  a  decision  as  to  which  class  an 
unknown  pattern  belongs  (27:39).  This  decision  is  usually  based  upon  the  application  of  decision 
functions  which  segment  the  feature  space.  Hyperplane  Classifiers  use  linear  decision  functions^  in 
the  form  of  hyperplanes,  to  partition  the  feature  space  while  Kernel  Classifiers  use  higher  order  de¬ 
cision  functions,  in  the  form  of  hyperspheres  or  hyperellipsoids,  to  partition  the  feature  space.  This 
chapter  begins  with  a  discussion  of  Hyperplane  Classifiers,  including  an  analysis  of  the  objective 
functions  used  to  implement  these  classifiers  as  neural  networks  and  their  parameter  update  equa¬ 
tions.  After  discussing  Kernel  Classifiers  in  general,  the  relationship  between  pattern  recognition, 
functional  interpolation  and  probability  estimation  are  examined.  This  chapter  concludes  with  the 
development  of  the  training  algorithms  used  to  implement  Kernel  Classifiers  as  neural  networks. 


3.2  Hyperplant  Classifiers 

3.2.1  Decision  Functions  Consider  the  two-dimensional  exemplars  representing  two  pattern 
classes  shown  in  figure  3.1.  As  can  be  seen,  the  two  patterns  can  be  separated  by  a  line  drawn  in 


Figure  3.1.  Linear  Decision  Function(27:40) 


the  feature  space.  The  general  equation  for  this  line  is 
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d{x)  —  tUlXl  +  102*2  +  O' 


(3.1) 


Here 

d{x)  is  the  linear  decision  function 
ti)  is  a  vector  containing  the  weights  or  scaling  coefficients 
X  is  the  pattern  vector  containing  the  feature  vsXues 
(T  is  an  offset  or  threshold 

Prom  this  figure,  d{x)  can  be  positioned  such  that  any  pattern  vector,  x,  belonging  to  class  A  will 
yield  a  positive  quantity  when  the  features  are  substituted  itxio  d{x)  while  any  pattern  belonging 
to  class  B  will  yield  a  negative  quantity  (27:39).  Thus,  cf(x)  ;an  be  considered  a  linear  decision 
function  since,  given  an  unknown  pattern  x,  d(x)  will  be  positive  for  class  A  and  negative  for  class 
B.  When  the  feature  space  has  K  dimensions,  the  ^^eneral  equation  for  the  linear  decision  function 
is  of  the  form 


d(x)  =  wixi  +  W2X2  +  . .  *wkxk  +  cr  (3.2) 

The  main  problem  associated  with  the  linear  decision  function  is  to  find  a  set  of  weights 
associated  with  the  decision  function  which  allows  the  feature  space  to  be  partitioned  in  a  manner 
which  separates  the  classes  (27:48). 

8.2»2  Network  Imptemeniaiion  The  characteristics  of  the  linear  decision  functions  can  be 
modeled  as  a  neural-type  element  by  assigning,  to  the  neural  element,  the  hyper-sigmoidal  transfer 
function 


y(x)  =  [l  +  e-'^(®)]-i  (3.3) 

where 


K 

*=i 

This  nonlinear  function,  as  shown  in  figure  3.2,  is  a  nondecreasing  function  in  which  the  output 
for  ^(x)  <  0  is  less  than  1/2  while  for  <^(x)  >  0  the  output  is  greater  than  1/2.  This  model 
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Figure  3.2.  Sigmoidal  TVansfer  Function 


is  based  on  the  Rosenblatt’s  perceptron  introduced  in  1957.  Again,  as  with  the  linear  decision 
function  described  above,  the  main  problem  associated  with  the  perceptron  is  to  set  the  weights, 
Wfii  and  offset,  cr,  such  that  the  feature  space  is  partitioned  to  allow  proper  classification  (19:50). 
These  parameters  can  be  established  by  performing  a  gradient  descent  using  a  method  known  as 
backpropagation. 

S,t3  Network  Training  The  training  method  most  commonly  used  with  this  perceptron- 
based  Hyperplane  Classifier  is  the  method  of  backpropagation.  In  this  method,  the  network  is 
presented  a  pattern  vector  and  allowed  to  produce  its  own  output.  This  output  is  then  compared 
with  the  desired  output  using  some  predefined  objective  function.  If  there  is  no  major  difference, 
then  no  learning  takes  place.  Otherwise  the  weights  and  offsets  for  each  node  are  changed  in  a  man¬ 
ner  which  optimizes  the  classification  objective  function  (22:322).  This  optimization  is  performed 
via  an  incremental  gradient  descent  on  the  surface  of  the  weight  space  whose  height  at  any  point  is 
equal  to  a  measure  of  the  performance  of  the  classification  objective  function  (22:322).  The  three 
main  types  of  classification  objective  functions  are  Mean  Square  Error  (MSB),  Cross  Entropy  (CE) 
and  Classification  Figure  of  Merit  (CFM)  (28:217). 

S.2,8,1  Mean  Square  Error  (MSE)  Objective  Function  The  Mean  Square  Error  (MSE) 
objective  function  seeks  to  minimize  the  mean  squared  error  between  the  network’s  actual  output 
and  the  desired  output  for  each  classification  node  in  the  output  layer  (28:217).  Suppose  a  particular 
pattern  recognition  problem  had  N  classes  and  the  network  was  developed  such  that  each  node  in 
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the  output  layer  represented  only  one  of  the  N  classes.  Let  dn  be  the  desired  output  for  the 
node  for  a  given  input  pattern.  The  MSE  then  is  defined  as 

MSE^^f^iyn-dnf  (3.5) 

n=:l 

Usually  during  training,  dn  is  taken  to  be  1  for  the  node  responsible  for  a  class  and  the  0  for 
the  rest  of  the  output  nodes.  This  objective  function  was  the  first  to  be  implemented  in  the 
study  of  artificial  neural  networks  and  is  the  most  widely  used  of  the  three  objective  functions. 
Any  network  trained  using  this  objective  function  will  make  a  classification  based  on  the  Bayes’ 
optimal  descriminant  (21).  However,  there  are  certain  properties  of  this  function  which  don’t 
permit  accurate  classifications  in  all  cases  (3).  As  shown  in  Appendix  A,  there  are  certain  areas  in 
the  feature  space  in  which  the  mean  square  error  is  higher  for  a  correct  classification  than  for  an 
incorrect  classification  (28:218),  This  implies  there  are  certain  areas  of  the  feature  space  in  which 
backpropagation  according  to  the  MSE  function  mayTail  to  separate  the  classes  correctly. 

A  network  implemented  using  the  MSE  objective  function  will  have  its  parameters  set  to 
minimize  the  MSE.  Thus,  the  general  update  equation  for  the  network’s  parameters  will  have  the 
following  form: 


V). 


.  ^  ,dMSE. 

~  —  4JI.  _  *■*«  -  ) 

dwi  ^ 


q  =  wj  -  7?( 


(3.6) 


Here,  r;  is  a  constant  which  controls  the  update  rate.  The  incremental  update  equations  for  each 
of  the  parameters  of  the  net  work  shown  in  figure  3.3  are  derived  in  Appendix  B  and  summarized 
below.  The  update  equation  for  a  weight  linking  node  M  in  layer  2  to  a  node  N  in  layer  3,  Is 


-  dAr)yiv(l  -  yN)yM 


(3.7) 


while  the  update  rule  for  the  offset  of  the  node  N,  (Tiv,  in  layer  3  is 


=  viVN  -  dAr)yAr(l  -  Vn)  (3.8) 

The  update  equation  for  a  weight  linking  node  L  in  layer  1  to  node=M  in  layer  2,  is 

N 

^LM  =  '^LM  -  V  53(yn  -  d„)yn{l  “  yn)wMnyM0-  "  yM)yL  (3.9) 

n=:l 
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Figure  3.3.  Two  Hidden  Layer  Sigmoidal  Network  Topology 

while  the  update  equation  for  the  offset  of  node  M  in  layer  2,  (TAf ,  is 

N 

-  4)j/n(l  “  yn)«'MnyM(l  -  Vm)  (3.10) 

n=l 

The  update  equation  for  a  weight  linking  node  K  in  layer  0  to  node  L  in  layer  l^WKiyis 

N  M 

^KL  =  ^KL  -  ^  -  dn)yn{l  ~  yn)(X^  Wmnym(.l  -  ym)wimyL(l  “  yL)yK]  (3.11) 

n=l  m=l 

while  the  update  equation  for  the  offset  of  node  L  in  layer  1,  is 

N  M 

crt  =  (Tl-ri  ^(y„  -  d„)y„(l  -  y„)[]^  w^nymi^  -  ym)wLmyLil  -  yi)]  (3.12) 

n=l  m=l 

S, 2.3.2  Cross  Entropy  (CE)  Function  The  Cross  Entropy  (CE)  Function  considers 
the  actual  value  of  an^  output  node  as  the  probability  that  the  ideal  binary  output  state  of  the 
node  is  a  1.  The  CE  function  seeks  to  minimize  the  difference  between  the  actual  output ,  Pm  and 
the  ideal  output,  by  minimizing  the  cross  entropy  between  the  actual  and  desired  probability 
density  functions  driving  the  output  nodes  (28:217).  That  is 


(3.13) 


CE  =  ^[dn  log(yn)  +  (1  -  dn)  log(l  -  y„)] 
n=:l 

Again  during  training,  dn  is  usually  set  to  1  for  the  node  responsible  for  the  correct  class  and 
to  0  for  the  rest  of  the  output  nodes.  As  shown  in  Appendix  A,  the  CE  function  is  also  characterized 
by  some  areas  in  the  feature  space  in  which  the  CE  is  greater  for  an  correct  classification  than  for 
an  incorrect  classification  (28:218).  This  implies  there  are  certain  areas  of  the  feature  space  in 
which  backpropagation  according  to  the  CE  function  may  fail  to  separate  the  classes  correctly. 

A  network  implemented  to  use  the  CE  function  as  the  classification  objective  function  will 
have  its  parameters  set  to  minimize  the  CE.  Thus,  the  general  update  equation  for  the  network’s 
parameters  will  have  the  following  form: 

=  (3.14) 

Here,  ??  is  a  constant  which  controls  the  learning  rate.  The  incremental  update  equations  for  each 
of  the  parameters  of  the  network  shown  in  figure  3.3  are  derived  in  Appendix  B  and  summarized 

below.  The  update  equation  for  a  weight  linking  node  M  in  layer  2  to  a  node  N  in  layer  3,  wmN}  Is 

^MN  =  ^MN  +  -  yN)yM  (3.15) 

while  the  update  rule  for  the  offset  of  the  node  N,  o*//,  in  layer  3  is 

^N  =  <^N  +  VidN  -  yN)  (3.16) 

The  the  update  equation  for  a  weight  linking  node  L  in  layer  1  to  node  M  in  layer  2,  wiMi  is 

N 

=  ^LM  +  V  Y'M"  ~  yn)«'AfnJ/A/(l  -  yM)yL  (3.17) 

n=l 

while  the  update  equation  for  the  offset  of  node  M  in  layer  2,  (rjvf,  is 

N 

=  -  l/n)wMnyAf(l  -  J/w)  (3.18) 

n=l 

The  update  equation  for  a  weight  linking  node  K  in  layer  0  to  node  L  in  layer  1,  wkl,  is 
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(3.19) 


N  M 

^KL  =  ^KL  +  »?  -  yn)[Y^  Wmnymil  “  ym)iy£mj/i(l  “  J/£)yK] 

n=l  m=l 

while  the  update  equation  for  the  offset  of  node  L  in  layer  1,  cr is 

N  M 

<^L=<^L+y  ^i^r.  -  yn)[X^  Wmnl/mCl  “  ym)«'£myii(l  “  yii)l  (3.20) 

n=l  m=:l 

S»2.3,3  Classification  Figure  of  Merit  (CFM)  Objective  Function  The  Classification 
Figure  of  Merit  (CFM)  objective  function  was  introduced  by  Waibel  to  minimize  the  classification 
errors  due  to  the  characteristics  of  the  MSE  and  CE  objective  functions  (28).  This  CFM  function 
does  not  consider  the  notion  of  an  ideal  output  during  training.  This  function  is  merely  concerned 
v/ith  forcing  the  correct  node  to  be  the  maximum  output  node  for  the  correct  input  features.  The 
CFM  objective  function  first  compares  the  activation  level  of  each  of  the  output  nodes  to  the 
output  node  which  should  have  the  highest  activation.  The  CFM  then  applies  a  sigmoidal  function 
to  differences  between  the  activation  levels  for  each  of  the  nodes  as  follows: 

1  ^ 

CFM  =  -^  S  +  (3-21) 

n=l»/c 

where  6n  =  yc-  Vn 

yc  =  response  of  the  correct  node 
Vn  =  response  of  the  incorrect  node 
N  =  total  number  of  output  nodes  or  classes. 
a  =  sigmoid  scaling  parameter. 

0  sigmoid  discontinuity  parameter. 

C  =  sigmoid  lateral  shift  parameter. 

The  application  of  the  sigmoidal  function  keeps  the  network  from  trying  to  produce  ideal  values 
as  the  CFM  function  yields  decreasingly  marginal  updates  for  increasingly  ideal  output  patterns. 
Also,  in  order  to  keep  the  network  from  attempting  to  learn  extreme  statistical  outliers  of  a  class, 
the  CFM  seeks  to  apply  decreasing  marginal  penalties  for  increasingly  bad  misclassifications  (28). 
As  shown  in  Appendix  A,  these  characteristics  may  allow  the  CFM  objective  function  to  reduce 
the  areas  within  the  feature  space  in  which  the  CFM  is  higher  for  an  incorrect  response  than  for  a 
correct  response. 
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A  network  implemented  using  the  CFM  objective  function  will  have  its  parameters  set  such 
that  the  CFM  objective  function  is  maximized.  Thus,  the  general  form  for  the  update  equation  of 
any  parameter  Wj  of  the  network  will  be 


wf  =  Wj  +r]{ 


dCFM. 
dwj  ^ 


(3.22) 


Again,  t;  is  a  constant  which  controls  the  learning  rate.  The  incremental  update  equations  for  the 
parameters  of  the  network,  shown  in  figure  3.3,  are  derived  in  Appendix  B  and  summarized  below. 
The  update  equation  for  a  weight  linking  node  M  in  layer  2  to  an  incorrect  node  N  in  layer  3,  WMNi 
is 


^MN  =  ^MN  -  -  ^Ar)yw(l  -  yiv)j/Jlf 

while  the  update  equation  for  the  offset  of  the  incorrect  node  N,  is 


(3.23) 


-  ZN)yN{l  -  Vn)  (3.24) 

The  update  rule  for  the  weight  linking  node  M  in  layer  2  to  the  correct  node  C  in  layer  3,  wmC}  Is 

N 

^MC  =  '^MC  +  ^  ^nO--Zn)ycO--yc)yM  (3.25) 

while  the  update  rule  for  the  offset  of  the  correct  node  C,  crc,  in  layer  3  is 

N 

=  +  y  Y  -  ^n)ycO- -  yc)  (3.26) 

The  the  update  equation  for  a  weight  linking  node  L  in  layer  1  to  node  M  in  layer  2,  wiMy  is 


iV 

‘^LM  =  '^LM+‘n  X]  ^n(l--?n)[yc(l-yc)«'Afe-yn(l-yn)«'Mn]yAf(l-yA/)yL  (3.27) 

njtc 

while  the  update  equation  for  the  offset  of  node  M  in  layer  2,  a/a,  is 
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N 


=  13  ^n(l-'^n)Ml-yc)t«Afc-yn(l-yn)t«Mn]yAf(l-J/Af)  (3.28) 

W—ln/c 

The  update  equation  for  a  weight  linking  node  K  in  layer  0  to  node  L  in  layer  1,  wkL}  is 


w 


N  M 

KL  =  +  13  2n(l  -  ^n)[yc(l  -  yc)  23  ti'mc 

nsinjtc  yn— 1 

M 

-  yn(i  -  yn)  23  «'mn]ym(i  “  ym)wLmyL(i  “  yi,)yK 


m=l 


(3.29) 


while  the  update  equation  for  the  offset  of  node  L  in  layer  1,  trx,,  is 


N 


M 


=  H+  'l  23  ^«(l  -  ^")bc(l  -  2/e)  Xl 


W„ 


*1  —  1  11960 


m=l 


M 


2/n(l  “  Vn)  ^mn]ym(l  “  Vl) 


m=l 


(3.30) 


S.3  Kernel  Classifiers 

3.3,1  Decision  Functions  Consider  the  two-dimensional  exemplars  representing  the  two 
classes  of  patterns  shown  in  figure  3.4.  As  can  be  seen,  the  two  patterns  can  be  separated  by 
placing  variable  diameter  circles  around  the  data  points  corresponding  to  each  class.  The  place¬ 
ment  of  these  circles  corresponds  to  a  partitioning  of  the  feature  space  into  receptive  fields  with 
each  circle  responding  to  a  pattern  only  when  the  features  falls  within  its  radius.  The  general 
equation  for  a  decision  function  for  this  type  of  pattern  recognition  system  is  given  by 

K  K 

d{x)  =  23  Wjjx]  -  23  +  ’"if+i  (3.31) 

i=l  »=1 

for  a  K-dimensicnal  feature  vector  (27:50).  Again,  the  weights,  and  tyx+i  represent  the 

coefficients  of  the  decision  function.  This  decision  function  can  be  written  in  matrix  form  as 


d(x)  =  xAx'^  —  xB  +  c 


(3.32) 
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Figure  3.4.  Circular  Decision  Functions 


Where  =  the  transpose  of  a  K-dimensional  vector  containing  the  input  features. 

X  =  [xiyX2i^  ^»iXK] 

Also  A  is  a  K  by  K  diagonal  matrix  containing  the  coefficients  of  the  squared  input  features. 

wii  0  ...  0 

0  W22  . . »  0 

A  = 

:  :  0 
0  0  ...  wkk 

jB^  is  a  k-dimensional  vector  containing  the  weights  or  coefficients  for  linear  input  feature  terms. 

B'^  =  [wi,W2,...,wk] 

Finally,  C  is  a  constant. 

The  coefficients  of  the  A  matrix  determine  the  shape  of  the  decision  boundaries.  If  A  is 
the  identity  matrix,  the  decision  functions  become  hyperspheres.  When  A  is  positive  definite,  the 
decision  functions  become  hypcrcllipsoids  and  when  A  is  positive  semidefinite  the  decision  functions 
become  a  hyperellipsoid  cylinder  (27:52).  Again,  the  main  problem  associated  with  these  high  order 
decision  functions  is  to  find  a  set  of  coefficients,  or  weights,  associated  with  the  decision  function 
which  allows  the  feature  space  to  be  partitioned  in  a  manner  which  separates  the  classes  (27:48). 
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S,S.2  Network  Implemeniaiion  The  characteristics  of  the  hyperspherical  decision  functions 
can  be  modeled  as  a  neural  type  element  by  assigning,  to  the  neural  element,  the  gaussian  transfer 
function 


where 


2/(5)  = 


(3.33) 


=  (3«) 

k=l 

Here 

Xk  =  dimension  of  the  input  pattern  vector  x 
Wk  =  k^^  dimension  of  the  weight  vector  w 
(Tk  =  spread  or  threshold  in  the  k^^  direction 

The  only  task  left  is  to  determine  how  to  architect  the  network  to  partition  the  feature  space  with 
these  neural  elements  and  perform  a  task  of  pattern  recognition.  This  architecture  can  be  derived  by 
applying  the  theory  of  approximating  multivariate  functions  using  Radial  Basis  Functions  (RBFs). 

3,3.S  Functional  Approximation  According  to  Powell  (17),  the  real  multivariable  interpo¬ 
lation  problem  is,  given  P  different  points  {xp\p  =  1,2,. ..,P)  in  a  K-dimensional  space,  and  P 
real  numbers  (dp\p  —  1,2, ...,P),  determine  a  function,  f{x)  from  into  3i  that  satisfies  the 
interpolation  conditions 


/(5p)  —  dp  (3.35) 

for  (p  =  1,2, .  ..,P).  This  function,  /(5),  can  be  decomposed  into  a  linear  combination  of  radial 
basis  functions. 


p=i 


(3.36) 


where 


X  €  of  and  p=  1,2, ...,P 
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Figure  3.5.  One  Dimensional  Radial  Basis  Function 


Xp  =  scaling  coefficient 

The  II  ...  II  is  usually  taken  to  be  the  Euclidean  Norm  while  the  Wp  are  the  centers  of  the  basis 
functions  (4:2).  A  radial  basis  function  is  a  function,  such  as  the  gaussian  shown  in  figure  3.5,  which 
is  symmetric  in  all  radial  directions  and  approaches  zero  as  the  distance  from  the  center  increases. 
Figure  3.6  shows  the  reconstruction  of  a  periodic  square  wave  from  a  linear  combination  of  these 
gaussian  radial  basis  functions. 

This  functional  approximation  can  be  implemented  with  a  neural  network  architecture  if  one 
considers  the  task  of  pattern  recognition  as  a  functional  mapping  from  the  set  of  data  points  to  the 
output  of  the  network.  Suppose  a  set  of  P  exemplars  characterize  the  pattern  recognition  problem. 
That  is,  sampling  of  the  environment  has  led  to  P  data  points  for  which  the  desired  classification 
of  each  data  point  is  known.  Further,  suppose  that  each  exemplar  is  a  K-dimensional  vector. 

x  =  [xi,X2y..-iXK]  (3.37) 

Thus  the  set  of  P,  K-dimensional  exemplars  characterize  the  pattern  recognition  problem.  Now, 
suppose  that  this  set  of  exemplars  can  be  classified  into  M  distinct  classes.  This  classification  can 
be  thought  of  as  a  mapping  from  a  K-dimensional  feature  space,  where  the  exemplars  reside,  into 
an  M-dimensional  space  where  the  classification  takes  place.  Let  the  desired  classification  vector  of 
a  given  exemplar,  say  Xp,  be  labeled  as  dm*  If  the  classification  problem  is  considered  as  a  mapping 
problem,  a  function  needs  to  be  determined  that  produces  the  following  result  for  each  of  the  P 
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Figure  3.6.  Square  Wave  Reconstructed  Via  Radie'  Basis  Functions 


exemplar  vectors,  Xpi 


fi^p)  (3.38) 

This  function,  once  found,  will  produce  the  desired  mapping  from  the  exemplar  points  to  their 
classifications.  Applying  the  theory  of  approximating  multivariate  functions  with  a  set  of  radial 
basis  functions,  this  function-finding  problem  can  be  modeled  as  a  real  multivariable  interpolation 
problem.  Using  the  gaussian  function  as  the  radial  basis  function,  the  approximation  can  then  be 
written  as 


f(^p) 


-CL  ‘='^1 


P=1 


(3.39) 


This  approximation  can  be  implemented  as  a  neural  network  architecture  as  shown  in  figure  3.7 
where  the  Ap’s  are  implemented  as  weights,  tu/m,  linking  the  nodes  in  the  hidden  layer,  layer  1, 
to  the  nodes  in  the  output  layer,  layer  2.  Here,  the  nodes  in  the  hidden  layer  have  the  gaussian 
radial  basis  function  as  their  transfer  function.  The  nodes  in  the  output  layer  compute  an  linear 


3-13 


combination  of  outputs  from  the  nodes  in  the  hidden  layer.  Thus,  the  overall  mapping  function, 
for  a  single  input  pattern,  takes  the  form  of 

L 

ym  =  '^Wlmyi  (3.40) 

1=1 

where 


(3.41) 


Once  this  network  is  established,  the  outputs  can  then  be  considered  as  a  mapping  from  the  input 
space  to  the  output  space.  This  mapping  can  also  be  considered  a  probability  density  estimation, 
via  the  technique  of  Parzen  Windows,  of  the  input  pattern  given  a  particular  output. 

8.S.4  Density  Esiimaiion  The  task  of  pattern  recognition  is  often  considered  as  a  problem 
of  assigning  an  unknov;n  sample,  say  5,  to  one  of  J  classes.  Bayes*  rule  for  this  classification 
problem,  called  Bayes*  optimal  discriminant  rule,  assigns  x  to  class  i  if 


P(xlGi)P{Gi)  >  P(x/Gj)P{Gj) 


(3.42) 
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for  all  i  ^  j  In  order  to  implement  this  rule,  the  underlying  probability  density  functions  (pdf^s) 
for  P{xlGj)  and  P{Gj)  for  all  J  classes  must  be  known.  One  method  of  estimating  these  density 
functions  is  to  group  the  elements  of  a  given  class  into  a  histogram.  The  problem  with  this  method 
is,  if  the  rectangular  cells,  or  bins,  into  'which  the  data  is  grouped  are  too  small,  the  estimate  may 
not  be  smooth.  If  the  rectangular  cells  are  too  big,  the  fine  details  of  the  distribution  may  be  lost. 
These  smoothness  problems  can  be  overcome  by  using  the  method  of  estimating  density  functions 
through  Parzen  Windows  or  kernel  estimators  (8:162). 

As  shown  in  figure  3.8,  the  Parzen  Window  estimate  of  the  density  function,  P{x/Gj)i  solves 


Figure  3.8.  Parzen  Window  PDF  Estimation  (8:164) 


this  smoothness  problem  by  assuming  that  each  value  of  the  data,  occurring  in  the  sample  set,  also 
raises  the  probability  of  any  value  occurring  close  to  that  value  of  the  data.  By  centering  a  kernel 
function  at  each  data  point,  the  final  value  of  the  estimate  can  be  obtained  by  summing  together  all 
the  contributions  from  each  value  of  the  sample  data  (8:162).  That  is  the  Parzen  Window  estimate 
has  the  form 


1  1  _  _ 

=  (3.43) 

where 

K  =  the  number  of  dimensions 

Nj  =  the  number  of  data  points  in  class  J 
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ft  =  a  function  of  Nj  referred  to  as  the  window  width. 

<j>{x)  =  the  kernel  function. 

To  implement  this  estimation,  an  appropriate  kernel  function  and  window  width  must  be  selected. 
As  shown  in  Appendix  C,  the  gaussian  radial  basis  function  of 


^(||x  —  Zn\\)  =  (27r(T^)  ^ ^  1  (3.44) 

can  be  used  as  the  kernel  function  for  a  Parzen  Window  estimation.  In  this  case,  the  equation  for 
the  density  becomes 


i=i 

For  a  two  class  problem,  with  the  ratio  of  the  number  of  sample  points  in  each  class  to  the 
total  number  of  sample  points  reflecting  the  apriori  probabilities,  the  classification  rule  is  to  assign 
X  to  group  I  if 


!!^p(i/Gi)>^Pix/Gj) 

Substituting  for  the  estimated  densities  provides  the  classification  rule 


(3.46) 


(3.47) 


This  reduces  to 


i=i 


E 


K  i*k-*kjr 


i=l 


(3.48) 


If  the  spreads,  or  window  widths,  are  the  same  for  each  class,  aj  ^  ex jy  then  equation  3.48  becomes 
a  sum  of  gaussian  radial  basis  functions. 


Nt  x^K 
t=i 


^-1 


Nj  r^K  (xfc-xfc,-)^ 
i=i 


(3.49) 
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These  equations  have  been  implemented  by  Specht  as  Probability  Neural  Networks  (PNNs) 
with  the  topology  shown  in  figure  3.9  (24).  In  this  network,  the  hidden  layer  weights,  or  centers 


of  the  gaussian  radial  basis  functions,  are  set  to  match  the  features  of  each  of  the  training  vectors. 
The  classification  nodes  in  the  output  layer  are  connected  only  to  the  nodes  in  the  hidden  layer 
which  belong  to  their  class.  These  output  layer  classification  nodes  implement  equation  3.49  by 
forming  the  simple  sum  of  the  outputs  from  the  hidden  layer  nodes  in  their  class.  In  this  type  of 
network,  the  <t’s  are  usually  chosen  on  a  trial  and  error  basis. 

The  only  difference  between  the  PNN  network  and  the  RBF  Kernel  Classifier  network  is 
that  the  classification  nodes  in  the  output  layer  of  an  RBF  network  are  connected,  via  weighted 
interconnections,  to  all  the  nodes  in  the  hidden  layer.  Since  it  has  been  shown  (21)  that  any 
neural  network  that  has  its  parameters  set  to  minimize  the  MSB  objective  function  will  operate 
as  a  Bayers  optimum  discriminant,  if  the  weights, u;,  and  sigmas  ,(r’s,  for  the  RBF  network  are 
established  via  minimization  of  the  MSB  objective  function,  the  RBF  network  should  approximate 
the  Bayes’  optimal  discriminant  function  without  the  trial  and  error  approach  to  setting  the  cr’s 
as  in  the  PNN.  Thus,  the  performance  of  PNN  network  shown  in  figure  3.9  should  approach  the 
performance  of  the  RBF  network  shown  in  figure  3.7  once  an  optimal  choice  of  a  is  made. 
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3,S.5  Network  Supervised  Training  As  with  the  Hyperplane  Classifier  network,  all  the  pa¬ 
rameters  for  the  Kernel  Classifier  network  can  be  established  by  minimizing  the  MSB  objective 
function,  incrementally,  via  backpropagation.  Again,  the  MSB  is  defined  as 

1 

(3.50) 

m=l 

The  general  form  of  the  update  equation  for  a  network  parameter  wj  then  becomes 


dMSE 


(3.51) 


Here,  ??  is  a  constant  which  controls  the  learning  rate.  These  update  equations  for  a  network  with 
the  topology  shown  in  figure  3.7  are  derived  in  Appendix  D.  For  a  weight  linking  node  L  in  Layer 
1  to  node  M  in  the  Layer  2  wlM}  the  update  equation  is 


^LM  =  -  dM)yL  (3.52) 

For  a  weight  linking  node  K  in  Layer  0  to  node  L  in  layer  1,  wkl^  the  update  equation  is 

^KL  =  ^KL-'nYl(y”*~  <^m)wLmyL  (3.53) 

m=l 

while  for  the  spread  of  node  L  in  layer  1  in  the  direction  of  node  K  in  layer  0,  (jkl  the  update 
equation  is 


^KL  “  ^  ~~  dm)^LmyL^  3  ^  (3.54) 

3,3,6  Network  Combined  Training  In  this  type  of  training,  the  hidden  layer,  layer  1,  weights 
(radial  basis  function  centers),  the  hidden  layer,  layer  1,  spreads  (radial  basis  function  sigmas),  and 
the  output  layer,  layer  2,  weights  are  set  separately.  That  is,  the  hidden  layer  weights  can  be  set 
by  any  of  the  following  rules: 

1.  .Nodes  at  the  Data  Points 

2.  Kohonen  Training 

3.  K-Means  Clustering 

4.  Center  at  Class-Cluster  Averages 
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The  hidden  layer  spreads  can  be  set  by  any  of  the  following  rules 

1.  Set  Sigmas  at  a  constant. 

2.  Set  Sigmas  at  P-Neighbor  Averages 

3.  Scale  Sigmas  by  Class  Interference. 

The  weights  linking  the  output  layer  to  the  hidden  layer  nodes  can  be  set  by  one  of  the  following 
rules: 

1.  Incremental  MSE  Minimization 

2.  Global  MSE  Minimization 

3.  PNN  Implementation 

3.3.6. 1  Nodes  at  the  Data  Points  In  this  training  algorithm,  the  hidden  layer  weights, 
or  centers,  of  the  radial  basis  functions  are  set  to  match  the  features  of  each  of  the  training  vectors. 
Suppose  there  are  P  pattern  vectors  where  each  vector  is  of  dimension  K.  The  vector  for  the 
pattern  vector  can  be  written  as 


Xp  —  \xp\^Xp2i . . . Xpj^]  (3.55) 

Setting  the  weights  to  match  the  exemplars  will  then  allow  P  hidden  layer  nodes  to  be  created 
where  the  output  of  the  node,  due  to  the  input  pattern  will  be  defined  from 


Vpi-e 


(3.56) 


Since  the  weights  match  the  exemplar  features,  the  weight  vector  for  the  radial  basis  function 
will  be  exactly  the  same  as  the  feature  vector  for  the  exemplar,  wi  =  xi.  This  allows  the  output, 
due  to  the  input  pattern,  for  the  radial  basis  function  to  be  written  as 


CK 

y/  =  e  *1 


(3.57) 


There  are  several  advantages  of  establishing  the  weights,  or  centers,  of  the  radial  basis  nodes, ^in 
this  manner.  First,  this  allows  the  direct  application  of  the  theory  of  approximating  multivariate 
functions  with  radial  basis  functions.  Second,  this  method  allows  a  direct  implementation  of  the 
Parzen  Window  probability  density  estimation  for  the  training  data.  Tliird,  the  computational  time 
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for  setting  the  layer  1  weights  is  negligible.  Finally,  each  radial  basis  function  node  is  guaranteed 
to  represent  a  particular  class  of  data  at  its  maximum  output. 

There  are  some  disadvantages  of  making  the  weights,  or  centers,  of  the  radial  basis  function 
nodes  match  the  exemplar  features.  First,  a  large  number  of  nodes  could  be  required  to  effectively 
partition  the  whole  feature  space.  Second,  the  computational  time  to  required  to  establish  the 
weights  in  the  output  layer  will  increase  significantly  as  the  number  of  nodes  increases.  Finally,  the 
weights,  of  centers  of  the  radial  basis  functions,  could  include  the  noise  associated  with  the  input 
patterns. 


8. 3, 6, 2  Kohonen  Training  The  Kohonen  Training  Algorithm  is  a  clustering  algorithm 
which  seeks  to  learn  the  underlying  probability  density  function  of  the  data  (19:64),  Using  this 
algorithm  to  set  the  weights,  or  centers,  of  the  radial  basis  functions  should  allow  a  radial  basis 
function  node  to  respond  strongly  to  similar  inputs.  Basically,  Kohonen  Training  calls  for  the 
establishment  of  a  rectangular  grid  of  nodes  as  shown  in  figure  3.10.  The  weights  for  these  nodes 


are  adapted  by  applying  a  training  vector  to  the  layer  and  computing  the  Euclidean  distance 
between  the  weights  for  each  of  the  nodes  and  the  input  vector  (19:65-68). 

K 

di  =  Y^{xpk  -  Wkif  (3.58) 

*=1 
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Notice,  this  distance  measurement  is  the  same  as  that  of  the  numerator  of  the  gaussian  transfer 
function  of  each  of  the  nodes  in  the  hidden  layer.  After  this  distance  calculation  is  made,  the  node 
whose  weights  are  nearest  to  the  features  of  the  input  patterns  are  updated,  along  with  nodes  in 
the  vicinity,  or  neighborhood  of  this  nearest  node,  according  to  the  equation 


^tt  =  Hf  +  -  ^ki)  (3.59) 

This  update  equation  serves  to  move  the  weights  of  each  of  the  updated  nodes  toward  the  input 
pattern  in  a  method  which  represents  the  vectorial  difference  between  the  weight  vector  and  the 
input  feature  vector  (19:67).  This  algorithm  of  presenting  an  input  pattern,  finding  the  node  with 
the  most  similar  weights  and  updating  that  node  and  its  neighbors,  is  repeated  over  a  specified 
number  of  iterations.  Once  the  Kohonen  layer  has  been  trained,  each  node  will  represent  clusters, 
or  pockets,  of  pattern  vectors. 

The  main  advantage  of  training  the  weights  in  this  manner  is  that  the  number  of  nodes  in  the 
layer  will  not  depend  explicitly  on  the  number  of  exemplars.  That  is,  there  can  be  far  less  nodes 
than  exemplars.  Also,  each  node  will  represent  more  than  one  exemplar  as  the  weights  are  trained 
to  represent  clusters  of  data. 

The  main  disadvantage  of  training  the  weights  in  this  manner  is  the  amount  of  time  it  takes 
to  train.  At  this  time,  there  is  no  formal  criteria  for  determining  when  the  weights  have  all  been 
adapted  to  represent  the  underlying  distribution  of  the  data.  Also,  the  weights  are  adapted  in  a 
manner  which  does  not  reflect  classification  of  the  data.  It  is  possible  for  a  node  in  the  Kohonen 
layer  to  respond  strongly  for  several  different  classes.  This  could  serve  to  hinder  the  training  of  the 
weights  in  the  output  layer.  Finally,  the  number  of  nodes  in  the  Kohonen  layer  is  arbitrary.  At 
this  time,  there  is  no  formal  method  of  predetermining  the  number  of  nodes  necessary  to  provide 
the  optimum  performance. 

S,3,6,3  K- Means  Clustering  The  K-Means  Clustering  Algorithm  is  a  method  of  train¬ 
ing  the  weights,  or  centers,  of  the  radial  basis  function  nodes  such  that  the  distance  from  all  points 
in  a  cluster  to  the  cluster  center  is  minimized  (27:94).  In  this  procedure,  the  number  of  radial  basis 
function  nodes  in  the  hidden  layer  is  preset  to  a  number  K.  The  weights  for  each  of  these  nodes  are 
initialized  to  match  the  features  of  the  first  K  pattern  vectors.  That  is  wi  =  x;  for  all  I  <  K.  All 
training  vectors  are  then  presented  to  the  network.  Each  vector,  Xp,  is  assigned  a  cluster,  denoted 
by  by  Xp  G  Jf  |j.Tp  — <  pp  — u>t||  for  all  i  =  1,2, .  ..,/<*  and  f  j.  Here,  the  norm  is  taken 
to  be  the  Euclidean  distance.  Notice  the  norm  is  the  same  as  the  numerator  in  the  gaussian  transfer 
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function  for  the  radial  basis  function  nodes.  This  means  each  new  pattern  vector  is  associated  with 
the  node  whose  center  is  the  closest  in  a  Euclidean  measure.  Once  all  patterns  have  been  assigned 
a  cluster,  the  new  cluster  center  weights  are  computed  as  the  average  of  the  features  of  the  pattern 
vectors  assigned  to  the  cluster.  That  is 

1 

—  IT  ^  ®nc  (3.60) 

Here,  Nc  is  the  number  of  pattern  vectors  assigned  to  the  cluster  and  Xnc  is  a  pattern  vector 
assigned  to  that  cluster.  Since  this  procedure  adapts  the  weights,  it  must  be  repeated  until  the 
weights  stabilize,  or  no  longer  adapt.  This  occurs  when  tZ)|(n  +  1)  =  wi{n)  for  all  cluster  centers. 

The  main  advantage  of  training  the  weights,  or  centers  of  the  radial  basis  function  in  this 
manner  is  that  the  number  of  nodes  does  not  depend  on  the  number  of  exemplars.  This  means 
there  can  be  many  more  exemplars  than  nodes,  with  each  node^s  weights  centered  at  the  average 
of  the  pattern  vector  features  associated  with  the  cluster.  Furthermore,  each  radial  basis  function 
node  will  now  be  able  to  represent  pattern  vectors  with  similar  features. 

The  main  disadvantages  are  that  the  number  of  radial  basis  function  nodes,  which  is  deter¬ 
mined  by  K,  the  number  of  clusters,  is  arbitrary  and  each  node  is  now  allowed  to  respond  strongly 
to  pattern  vectors  of  different  classes.  Also,  the  performance  of  the  algorithm  is  dependent  on  the 
number  of  clusters,  initial  location  of  the  clusters  and  the  properties  of  the  data  (27:95).  Finally, 
there  is  no  guarantee  that  the  algorithm  will  converge. 

S.S.6,4  Center  at  Class-Cluster  Averages  In  this  algorithm  the  weights,  or  centers  of 
the  radial  basis  function  nodes  are  allowed  to  adapt  themselves,  in  an  iterative  process,  to  the 
centers  of  clusters  of  pattern  vectors  of  the  same  class.  Furthermore,  this  algorithm  is  adaptive  in 
the  sense  that  the  number  of  nodes  does  not  need  to  be  preselected.  The  distribution  of  the  data  will 
determine  the  required  number  of  radial  basis  function  nodes.  In  this  algorithm  a  cluster  radius,  R, 
is  first  preset  and  the  network  begins  with  one  node  whose  weights  match  the  first  pattern  vector. 
This  node  is  also  set  to  respond  to  the  class  of  the  first  pattern  vector.  A  new  pattern  vector,  Xp, 
is  then  applied  to  the  network.  The  cluster  assignment  rule  is  x  G  Sj  if  ||x  —  u)j||  <  ||x  —  u)f||  <  R 
and  the  class  of  x  is  the  same  as  Sj.  If  this  relation  isn’t  true  then  a  new  node  is  added  such  that 
the  weights  and  cla^  of  the  new  node  match  that  of  the  new  input  pattern.  If  this  relation  is  true, 
then  the  weights  of  the  cluster  center  are  adapted  to  the  new  average  by 
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Wj{t  +  1)  =  Wj{t)  +  — 


(3.61) 


This  process  of  presenting  a  new  pattern  vector  to  the  network,  checking  to  see  if  this  new  pattern 
vector  can  be  associated  with  an  existing  cluster  and  adjusting  the  cluster  center  or  adding  a  new 
cluster  center  will  continue  until  all  exemplars  are  tested.  Since  it  is  possible  for  exemplars  to 
become  ”  uncovered”  during  the  update  of  cluster  centers,  the  algorithm  is  repeated  until  no  new 
nodes  are  added. 

The  main  advantages  of  this  algorithm  are  that  the  number  of  radial  basis  function  nodes 
does  not  need  to  be  selected  beforehand  and  each  node  will  respond  strongly  to  only  one  class. 

The  main  disadvantage  of  this  algorithm  is  that  the  association  radius,  or  vigilance  parameter 
R,  must  be  selected  arbitrarily. 

S.S.6,5  Set  Sigmas  ai  Consiani  In  this  algorithm,  the  sigma,  or  spread,  for  each  radial 
basis  function  node  is  preset  to  a  constant,  C.  Under  this  condition,  the  output  for  the  radial 
basis  function  node  due  to  the  pattern  vector  Xp  becomes 

yp,  =  (3.62) 

If  the  weight  vectors  had  been  previously  set  to  match  the  features  of  the  exemplar  vectors,  the 
network  then  calculates  a  Parzen  Window  estimate  of  the  probability  distributions  of  the  data. 

The  main  disadvantage  of  training  the  sigmas  in  this  manner  is  that*  since  the  constant  is 
preset  and  not  changed,  there  is  no  way  of  determining  if  the  entire  feature  space  is  partitioned. 

Set  Sigmas  ai  P-Ntighbor  Averages  In  this  algorithm,  the  sigmas  for  each 
radial  basis  function  node  are  allowed  to  vary  according  to  a  distance  metric  between  their  weights, 
or  centers,  and  the  weights  or  centers  of  their  P  nearest  neighbors.  That  is,  after  the  weights,  or 
centers,  of  each  radial  basis  function  node  is  set,  the  Euclidean  distance  between  the  center  of  each 
radial  basis  function  node  and  its  neighbors  are  calculated.  For  example,  the  distance  between 
radial  basis  function  node  i  and  radial  basis  function  node  j  is 

K 

dij  =  Y^{wkj  -  Wkif  (3.63) 

k=l 
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Ftom  these  calculations,  the  P  radial  basis  functions  having  the  smallest  distance,  dip,  are  then 
used  to  set  the  sigma,  or  spread,  (<Ti),  for  the  radial  basis  function  by  the  following  equation: 


(Ti  = 


4E4 


P=1 


(3.64) 


This  equation  makes  (Ti  equal  to  the  root  mean  square  of  the  sum  of  distances  between  the  center 
of  radial  basis  function  and  its  P  nearest  neighbors  (13:137). 

The  main  advantages  of  setting  the  cr’s  in  this  manner  are  that  each  a  can  be  different  for 
each  node  and,  since  each  cr  is  a  function  of  the  separation  between  node  centers,  the  feature  space 
will  usually  be  completely  partitioned. 

The  main  disadvantage  of  setting  the  <t^s  in  this  manner  is  that  P  must  be  determined 
beforehand.  If  P  is  too  small,  the  cr^s  will  be  small  and  the  feature  space  will  not  be  covered 
adequately.  If  P  is  too  large,  the  <r^s  will  allow  too  much  overlap  between  patterns  of  different 
classes.  This  could  result  in  a  node  responding  too  strongly  to  more  than  one  class. 


S. 3,6.7  Scale  Sigmas  by  Class  Inierference  In  this  algorithm,  the  are  adjusted, 
from  a  preset  constant,  to  prevent  the  radial  basis  function  nodes  from  responding  too  strongly 
from  pattern  vectors  of  different  classes.  In  order  for  this  algorithm  to  work,  each  radial  basis 
function  node  must  be  assigned  the  responsibility  for  responding  to  only  one  class  for  the  training 
data.  This  can  be  done  by  setting  the  weights  using  the  Nodes  at  Data  Points  or  Center  at  Class- 
Cluster  Averages  Algorithms  previously  discussed.  After  the  weights  are  set,  this  algorithm  then 
presents  an  exemplar  pattern  Xp  and  calculates  the  output  for  each  radial  basis  function  node  by 


ypi  =  e“3i»  (3.65) 

If  the  output  for  that  node  is  above  some  preset  threshold,  T,  and  the  node  is  not  assigned  to 
respond  to  the  same  class  as  that  of  the  pattern  vector,  Xp,  then  that  node^s  a  is  scaled  by  a 
constant  until  the  output  is  less  than  T.  That  is  if  ypi  >  T  then 


fff  =  (1  -  C)(rJ- 


This  process  is  then  repeated  for  each  pattern  vector  in  the  training  set. 


(3.66) 
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The  main  advantage  of  setting  the  cr^s  in  this  manner  is  that  each  node  is  guaranteed  to 
respond  to  one  and  only  one  class  for  the  training  data  within  some  threshold  T.  Another  advantage 
is  that  each  node  can  now  have  a  separate  <7. 

The  main  disadvantage  of  setting  the  <t’s  in  this  manner  is  that  if  the  pattern  vectors  are 
very  close  together,  via  the  Euclidean  distance  measurement,  then  each  node  will  lose  its  ability  to 
generalize. 


3.3.6. 8  Incremental  MSE  minimization  As  shown  in  appendix  D,  the  update  equation 
for  the  weight  linking  node  L  in  layer  1  to  node  M  in  layer  2,  wim  is 

“  ^LM  ~  ViVM  *”  <^Af)yL  (3.67) 

These  parameters  can  be  updated  through  backpropagation  even  though  the  other  network  param¬ 
eters  have  been  preset.  However,  this  process  can  still  take  many  iterations  to  converge. 

3. 3.6.9  Global  MSE  minimization  As  shown  in  appendix  D,  if  the  weights  and  sigmas 
(spreads)  of  the  first  layer  have  been  established,  the  weights  linking  nodes  layer  1  to  nodes  in  layer 
2  can  be  established  by  a  global  minimization  of  the  MSE  function  over  all  training  patterns  (23). 
When  trained  in  this  manner,  the  update  equation  for  a  weight  linking  node  B  in  layer  1  to  node 
D  in  layer  2  is 


W  =  (3.68) 

Here,  W  is  an  L  by  M  matrix  containing  the  weights  linking  the  nodes  in  the  hidden  layer  to  an 
nodes  in  the  output  layer.  That  is 


Wu 

Wl2 

•  •  •  WiM 

W21 

^22 

•  ■  •  W2M 

«^L2 

. . .  WLM 

(3.69) 


where  wlm  is  the  weight  linking  the  node  in  the  hidden  layer,  layer  1,  to  the  node  in 

the  output  layer,  layer  2.  The  M  matrix  is  an  L  by  L  matrix  containing  the  summation,  over  all 
patterns,  of  the  product  of  each  radial  basis  function  output,  for  a  given  input  pattern  and  the 
radial  basis  function  output  for  that  pattern.  That  is 
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M\i  Mi2  *  •  • 

M2X  M22  •  •  •  ^2L 

JWxri  Mi2  •  •  •  Mil 

where  Mm  =  J2p=:i  VpiVpB-  Also,  Y  is  a  P  by  L  matrix  containing  the  outputs  for  each  of  the  L 
radial  basis  functions  for  all  P  patterns.  That  is, 

yii  yn  •••  ViL 

Y  — 

^  ypi  yp2  •  •  •  ypL 

Finally  S  is  a  P  by  M  matrix  containing  the  desired  outputs  for  each  of  the  M  output  nodes  for  all 
P  patterns.  That  is, 

dll  di2  ...  diM 

d21  d22  *  •  •  d2M 

dpi  dp2  . . .  dpM 

This  method  only  works  for  matrices  that  do  not  become  singular  or  near-singular,  which  can 
happen  if  the  exemplar  data  points  used  to  center  the  radial  basis  functions  contain  redundant 
information.  If  they  do,  the  Singular  Valued  Decomposition  of  the  matrix  may  be  used.  Conversely, 
the  as  of  the  offending  nodes  may  be  adjusted  to  eliminate  the  redundancy. 

3,3,6.10  Probability  Neural  Network  (PNN)  As  shown  in  Chapter  3,  after  establishing 
the  parameters  for  the  nodes  in  the  hidden  layer,  layer  1,  a  PNN  can  be  constructed  by  connecting 
each  output  layer  node  to  the  hidden  layer  nodes  representing  the  output  layer  node^s  class.  In 
this  network,  the  weights  connecting  the  hidden  layer  nodes  in  layer  1  to  the  output  layer  nodes  in 
layer  2  are  set  to  T. 
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3.4  Summary 

This  chapter  discussed  the  general  operation  of  Hyperplane  and  Kernel  Classifier  neural  net¬ 
works.  The  objective  functions  used  to  implement  the  Hyperplane  Classifier  networks  were  then 
analyzed,  followed  by  the  development  of  the  equations  necessary  to  implement  these  classifiers 
as  neural  networks.  The  relationship  between  pattern  recognition,  functional  interpolation  and 
probability  density  estimation  were  then  presented  as  implementable  properties  of  Kernel  Classifier 
networks.  This  chapter  concluded  with  the  development  of  equations  implementing  these  classifiers 
as  neural  networks. 
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IV.  Software  Description 


4»1  Iniroduciion 

The  software  to  be  described  in  this  chapter  was  designed  according  to  an  object-oriented 
approach.  This  chapter  begins  with  a  description  of  the  data  structures  implemented  for  the 
software  and  concludes  with  a  brief  discussion  of  the  software  modules.  An  in-depth  description 
of  these  items,  along  with  the  mapping  of  the  training  algorithms  developed  in  chapter  3,  into 
software  functions,  is  given  in  Appendix  P. 

4-2  Approach 

Artificial  neural  networks  are  composed  of  nodes.  Each  node  has  associated  with  it  certain 
parameters  such  as  a  weight  vector,  an  offset,  a  transfer  function,  and  a  class  to  which  the  node 
responds.  The  main  difference  between  different  types  of  networks  is  the  way  in  which  the  nodes 
are  connected  to  one  another  and  the  method  of  setting  the  network  parameters.  Therefore,  in 
order  to  maximize  the  types  of  networks  which  could  be  configured,  the  only  entity  implemented 
as  an  object  was  the  node.  Each  node  was  then  assigned  the  following  attributes: 

(a)  weights  -  Wj 

(b)  sigmas  -  aj 
(b)  connections 

(d)  transfer-function 

(e)  class 

The  operations  that  can  be  performed  on  each  node  are  the  following: 

(a)  assign  transfer  function 

(b)  calculate  node  output 

(c)  initialize  node  weights  and  sigmas 

(d)  assign  a  node  to  a  class 

(e)  assign  a  node  to  be  connected  to  another  node 

(f)  update  (train)  weights  and  sigmas 

With  these  attributes  and  operations  a  variety  of  networks  can  by  formulated.  This  thesis 
implemented  just  the  feedforward  type  of  network  architecture.  However,  this  object  oriented 
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design  approach  will  allow  future  enhancements  to  more  complicated  networks  such  as  recurrent 
networks  and  higher-order  networks. 

4^3  Networks 

The  only  type  of  network  implemented  at  this  time  is  the  feed  forward  network.  A  feed 
forward  network,  as  shown  in  figure  4.1,  is  a  network  in  which  each  node  is  assigned  to  a  particular 
layer  and  receives  inputs  only  from  the  nodes  in  the  previous  layer.  Pattern  vectors  are  input  to 
the  network  via  the  nodes  in  layer  0.  These  nodes  have  the  identity  transfer  function  and  serve 
to  propagate  the  features  from  the  input  pattern,  across  the  internodal  weights  and  thresholds,  to 
nodes  in  layer  1.  The  layer  1  nodes  will  transform  these  inputs  into  internal  representations,  using 
their  assigned  transfer  functions,  and  transmit  these  representations,  via  the  internodal  weights 
and  thresholds,  to  the  nodes  in  layer  2.  This  process  of  transforming  the  data  and  propagating  the 
new  representation  will  continue  through  each  layer  of  the  network.  The  outputs  for  the  last-layer 
in  the  network  will  be  used  to  determine  the  classification  of  the  input  pattern. 
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Figure  4.2.  Software  Structure  Chart 


4.4  Siruciure 

The  software  implemented  in  this  thesis  consists  of  the  nine  modules  shown  by  the  structure 
chart  in  figure  4.2. 

4-4»^  NETMENU  This  module  is  the  overall  controlling  module  of  the  network.  It  provides 
the  user  interface  to  the  software  via  the  SUN  terminal  and  keyboard  and  calls  the  appropriate 
modules  to  execute  the  users  decisions. 

4-4-^  NETERROR  This  module  contains  the  functions  necessary  to  determine  the  net¬ 
work’s  classihcatibn  of  a  data  vector  and  the  error  performance  of  the  network. 

4.4*^  NETTRAIN  This  module  contains  the  functions  necessary  to  establish  the  network 
weights  via  the  following  training  procedures: 

a)  Nodes  at  the  Data  Points 

b)  Center  at  Class-Cluster  Averages 

c)  K-means  Clustering 

d)  TVain  via  Kohonen 

e)  Global  MSB  Minization 
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f)  Backpropagation  for  MSE,  CE  and  CFM  algorithms. 

g)  Probability  Neural  Networks 

Each  of  these  functions  accomplishes  its  training  routine  by  executing  the  specialized  functions 
contained  in  NETAUX.  NETTRAIN  also  contains  the  functions  necessary  to  establish  the  <r^s  for 
the  network  nodes  via  the  following  training  procedures: 

a)  Scale  Sigma  by  Class  Interference 

b)  Set  Sigma  According  to  P-Neighbor  Distance 

c)  Set  Sigma  to  a  Constant 

44*4  NETINPUT  This  module  contains  the  functions  necessary  to  load  the  training  and 
test  data  patterns.  This  data  may  be  loaded  from  separate  training  and  test  files  or  from  a  single 
file.  This  loading  of  training  and  test  patterns  may  also  be  accomplished  either  in  the  sequence 
listed  in  the  data  files  or  in  a  random  manner. 

^.^.5  NETINIT  This  module  contains  the  functions  which  allocate  memory  for  the  nodes 
and  data  records,  correct  node  weights  and  connections,  and  initialize  the  node  weights,  sigmas, 
transfer  functions  and  network  connections. 

44*^  NETSHOW  This  module  contains  the  output  functions  necessary  to  display  and  file 
the  performance  and  parameters  of  the  network. 

44^'^  NETOUT  This  module  contains  the  functions  necessary  to  compute  the  outputs  for 
each  node  in  the  network,  the  outputs  for  each  layer  of  a  feedforward  network,  and  the  output  for 
the  entire  network  due  to  a  given  input  pattern. 

44-^  NETAUX  This  module  contains  the  training  subfunctions  called  by  NETTRAIN. 
Appendix  F  contains  a  detailed  description  of  each  function  in  this  module. 

44-^  NETMATH  This  module  contains  the  mathematical  functions  used  by  the  various 
training  algorithms  within  the  module  NETTRAIN. 

4 >5  Implemeniaiion 

The  software  code  developed  for  this  thesis  and  implemented  under  these  modules  is  listed  in 
Appendix  G. 
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Ji.6  Summary 

This  chapter  provided  a  brief  overview  of  the  software  developed  for  this  thesis.  After  devel¬ 
oping  the  data  structures  implemented  in  the  softwared,  the  sectioning  of  the  software  into  modules 
was  outlined. 
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K  Data  Analysis 


5 A  Iniroduciion 

Using  the  software  developed  in  Chapter  4,  two  pattern  classification  problems  will  be  ana¬ 
lyzed.  The  first  problem  deals  with  the  classification  of  coded  digital  communication  signals  while 
the  second  problem  deals  with  classification  of  radar  platforms.  This  chapter  begins  by  discussing 
the  data  used  to  train  and  test  the  neural  networks  developed  to  classify  coded  digital  communica¬ 
tion  signals.  After  detailing  the  methods  used  to  train  various  networks  to  solve  this  classification 
problem,  the  results  of  the  training  and  testing  are  then  presented.  This  chapter  concludes  with  a 
discussion  of  the  data  used  to  train  and  test  the  neurahnetworks  developed  to  classify  radar  systems 
and  an  analysis  of  training  and  test  results. 


5.2  Communication  Signal  Characterization 

5.2 A  Data  Description  In  this  two-class  problem,  an  acousto-optic  correlation  system  was 
used  to  capture  correlation  signatures  of  spread  spectrum  signals  for  both  a  direct  sequence  and 
a  linear-stepped  frequency  hopped  signal.  Over  200. pattern  vectors  were  first  formed  by  sampling 
known  waveforms  at  1000  data  points.  After  averaging  consecutive  data  point  pairs,  thereby 
reducing  the  number  of  data  points  to  500,  the  peak  of  the  signals  were  identified  and  25  points 
on  each  side  of  the  peak  were  extracted.  These  50  data  points,  now  representing  a  50  dimensional 
feature  vector,  were  normalized  to  values  between  -1  and  1  by  dividing  each  dimension  by  the 
magnitude  of  the  largest  component.  One  hundred  feature  vectors  for  each  class  now  represented 
the  signature  for  the  direct  sequence  and  the  linear  stepped  frequency  hopped  encoding  schemes. 
Feature  vectors  representing  the  direct  sequence  signatures  were  then  assigned  to  class  1  and  feature 
vectors  representing  the  linear  stepped  frequency-hopped  signatures  were  assigned  to  class  2.  The 
final  data  set  contained  101  pattern  vectors  for  each  class;  with  each  pattern  vector  having  50 
dimensions. 

5.2.2  Testing  This  data  was  processed  using  both  Hyperplane  Classifiers  and  Kernel  Clas¬ 
sifier  networks.  The  parameters  for  each  of  these  networks  were  set  using  the  algorithms  developed 
in  Chapter  3.  The  training  data  for  each  of  the  classes  was  randomly  selected  for  each  network 
run.  For  each  test,  Sl  feature  vectors  from  each  class  were  used  to  train  the  network  and  a  different 
50  feature  vectors  from  each  class  were  used  to  test  the  network.  For  each  netv/ork,  there  were 
50  nodes  in  layer  0,  one  node  for  each  of  the  dimensions  of  the  input  data.  The  number  of  nodes 
in  the  final,  or  output,  layer  of  the  network  was  set  at  two  with  each  node  assigned  to  represent 
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one  of  the  two  classes.  The  number  of  hidden  layers  and  the  number  of  nodes  in  the  hidden  layers 
were  allowed  to  vary  according  to  the  parameters  of  the  network.  For  this  classification  problem, 
the  network  was  allowed  to  make  a  classification  based  on  which  node  in  the  output  layer  had  the 
higher  output. 

5.2,S  Hyperplane  Classifiers  The  Hyperplane  Classifier  networks  developed  for  this  classi¬ 
fication  problem  were  based  on  the  topology  shown  in  figure  3.3  with  each  network  consisting  of 
two  hidden  layers.  The  50  nodes  in  the  input  layer,  layer  0,  had  the  identity  transfer  function.  The 
number  of  nodes  in  the  first  hidden  layer,  layer  1,  was  set  at  18,  the  number  of  nodes  in  the  second 
hidden  layer,  layer  2,  was  set  at  ten,  and  the  number  of  nodes  in  the  output  layer,  layer  3,  was 
set  at  two.  The  nodes  in  each  of  these  layers  were  assigned  the  sigmoidal  transfer  function.  The 
parameters  for  each  of  the  nodes  were  trained  via  backpropagation  according  to  either  the  MSB, 
CE  or  CFM  objective  functions  discussed  in  Chapter  3. 

5,2,$.  J  MSB  Objective  Function  To  characterize  the  performance  of  Hyperplane  Clas¬ 
sifiers  trained  using  this  algorithm,  ten  different  sets  of  training  data  were  applied  to  the  network 
and  the  performance  of  the  network  was  measured  for  each  set  as  shown  Table  E.l.  Here,  a  correct 
response  for  the  training  data  is  defined  to  occur  when  the  output  for  the  correct  classification 
node  was  greater  than  .9  and  the  output  for  the  incorrect  classification  node  was  less  than  .1.  As 
shown  in  figure  5.1,  the  average  performance  of  the  network  converged  rapidly  until  about  15000  - 


Figure  5.1.  Performance  vs  Training  Iterations  for  MSB  Algorithm 


20000  iterations.  At  this  point,  the  categorization  performance  of  the  network  reached  90  percent. 
From  20000  -  30000  iterations,  the  categorization  performance  slowly  increased  to  the  final  average 
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of  97.16  percent.  The  robustness  of  the  network,  for  both  the  training  and  the  test  data,  was 
calculated  as  shown  in  Table  5.1. 


Table  5.1.  Robustness  Measure  for  MSB  Training 


%  Correct 

Training 

Test 

Avg 

Std 

97.16 

4.51 

79.7 

5.92 

5.2,3. 2  CE  Objective  Function  To  characterize  the  performance  of  Hyperplane  Classi¬ 
fiers  trained  using  this  algorithm,  ten  different  sets  of  training  data  were  applied  to  the  network  and 
the  performance  of  the  network  was  measured  for  each  set  as  shown  in  Table  E.2.  Here,  a  correct 
response  for  the  training  data  is  defined  to  occur  when  the  output  for  the  correct  classification  node 
was  greater  than  .9  and  the  output  for  the  incorrect  classification  node  was  less  than  .1.  As  shown 
in  figure  5.2,  the  average  performance  of  the  network  converged  rapidly  until  about  5000  -  10000 


iterations.  At  this  point,  the  performance  of  the  network  remained  relatively  stable.  From  10000 
-  15000  iterations,  the  categorization  performance  slowly  increased  to  the  final  average  of  100.00 
percent.  The  robustness  of  the  network  was  calculated  as  shown  in  Table  5.2. 

As  compared  to  the  networks  trained  to  minimize  the  MSE  objective  function,  the  networks 
trained  to  minimize  the  CE  objective  function  performed  at  about  the  same  level  of  categorization 
accuracy  but  converged  in  about  half  the  iterations  as  the  MSE  objective  function.  This  is  due  to 
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Table  5.2.  Robustness  Measure  for  CE  Training 


%  Correct 

Training 

Avg 

Std 

81.2 

4.62 

the  lack  of  the  term  2/n(l  ~  Vn)  in  the  CE  update  equations.  Since  the  maximum  value  for  this 
term  is  1/4,  the  parameters  for  the  MSE  network  are  adapted  much  more  slowly  than  that  of  the 
CE  network. 

5.2.8,S  CFM  Objective  Function  To  characterize  the  performance  of  Hyperplane  Clas¬ 
sifiers  trained  using  this  algorithm,,  ten  different  sets  of  training  data  were  applied  to  the  network 
and  the  performance  of  the  network  measured  for  each  set  as  shown  in  Table  E.3.  As  shown  in 
figure  5.3,  the  average  performance  of  the  network  converged  rapidly  until  about  30000  -  35000 


iterations.  At  this  point,  the  performance  of  the  network  remained  relatively  stable.  From  35000 
-  50000  iterations,  the  categorization  performance  slowly  increased  to  the  final  average  of  88.83 
%.  The  robustness  of  the  network  was  calculated  as  shown  in  Table  5.3.  As  expected,  the  catego¬ 
rization  performance  for  the  training  data  was  less  than  that  for  either  the  MSE  or  CE  objective 
functions.  However,  the  performance  for  the  test  data  proved  lower  than  that  of  either  the  MSE  and 
CE  objective  functions  for  this  data.  If  the  two  tests  having  the  lowest  categorization  performance 
on  the  test  data  are  removed,  the  categorization  performance  of  the  CFM  for  the  test  data  rises  to 
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Table  5.3.  Robustness  Measure  for  CFM  Training 


%  Correct 

Training 

Test 

Avg 

Std 

88.83 

5.37 

73.20 

7.93 

76.63  %  which  nearly  matches  that  of  the  MSB.  This  seems  to  indicate  the  pattern  space  does  not 
contain  pockets  of  data  which  would  cause  the  MSB  and  CB  algorithms  to  have  larger  classification 
errors  for  a  correct  response  than  for  an  incorrect  response. 

5,2,4  Kernel  Classifiers  The  networks  developed  to  categorize  this  data  are  based  on  the 
topology  shown  in  figure  3.7.  The  50  nodes  in  the  input  layer,  layer  0,  had  the  identity  transfer 
function  and  two  nodes  in  the  output  layer,  layer  2,  had  the  linear  transfer  function.  The  number 
of  nodes  in  the  hidden  layer,  layer  1,  was  a  function  of  the  algorithm  used  to  train  the  layer  1 
weights.  These  nodes  were  assigned  the  gaussian  transfer  function. 

5,2,4,!  Nodes  at  ihe  Data  Points  In  this  test,  the  performance  of  the  Kernel  Classifier 
network,  using  the  interpolation  theory  of  applying  the  nodes  at  the  data  points,  was  measured. 
The  weights,  or  centers,  for  the  nodes  in  layer  1  were  set  using  the  Nodes  at  Data  Points  algorithm 
and  the  sigmas  were  set  using  the  Scale  Sigmas  by  Class  Interference  algorithm.  The  weights  linking 
the  nodes  in  the  output  layer  to  the  nodes  in  the  first  layer  were  trained  via  global  minimization 
of  the  MSB  objective  function.  The  performance  of  the  network  was  then  analyzed  as  the  number 
of  nodes  in  the  hidden  layer  was  allowed  to  vary  from  10  to  50.  The  complete  data  are  shown  in 
Tables  B.4  and  E.5.  A  plot  of  this  performance  is  shown  in  figure  5.4.  As  expected,  as  the  number 
of  RBF  nodes  in  the  hidden  layer,  layer  1,  increased,  the  classification  performance  of  the  network 
increased  for  both  the  training  and  the  test  data.  The  maximum  performance  occurred  when  the 
number  of  nodes  matched  the  number  of  exemplars  at  102. 

The  overall  performance,  or  robustness,  of  the  network  when  layer  1  contained  102  nodes 
was  found  to  be  as  shown  in  Table  5.4.  As  shown  by  this  table,  setting  the  weights  of  the  layer 
1  nodes  at  the  training  data  points  allows  the  network  to  ’memorize^  the  training  data  while  still 
performing  relatively  well  on  the  test  data.  However,  this,  performance  is  at  the  expense  of  many 
layer  1  nodes. 


5-5 


Avg  %  correct 


Figure  5.4.  Performance  vs  Nodes  for  Nodes  at  Data  Points 


Table. 5.4.  Robustness  Measure  for  Nodes  At  Data  Points 


%  Correct 

Training 

Test 

Avg 

Std 

100.00 

0.00 

84.90 

3.94 
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Kohonen  Training  In  this  test,  the  ability  of  a  fixed  number  of  Kohonen  layer 
nodes  to  distribute  themselves  to  cover  the  pattern  space  was  measured.  The  weights,  or  centers, 
for  the  nodes  in  the  hidden  layer,  layer  1,  were  trained  using  the  Kohonen  Training  algorithm  and 
the  sigmas  set  using  the  Set  Sigmas  at  P-Neighbor  Averages  algorithm.  The  weights  linking  the 
nodes  in  the  output  layer  to  the  nodes  in  the  first  layer  were  trained  via  global  minimization  of  the 
MSE. 

With  P  arbitrarily  held  at  six,  the  performance  of  the  network  was  analyzed  as  the  number  of 
nodes  in  the  Kohonen  layer  was  increased  from  16  to  100.  The  complete  data  are  shown  in  Tables 
E.6  and  E.7.  As  shown  in  figure  5.5  as  the  number  of  nodes  increased,  the  ability  of  the  network  to 


Figure  5.5.  Performance  vs  Nodes  for  Kohonen  Training  with  Six  P  Neighbors 


categorize  the  training  data  increased.  Hov’cver,  the  ability  of  the  network  to  categorize  the  test 
data  decreased.  This  is  due  to  the  fact  that  as  the  number  of  nodes  is  increases,  the  distance  to  the 
six  nearest  neighbors  decreases.  Thus,  the  network  loses  its  ability  to  generalize.  The  decrease  in 
the  training  performance  for  100  nodes  was  due  to  two  of  the  networks  converging  to  a  performance 
of  less  than  85%.  If  these  two  tests  are  removed  from  the  performance  calculations,  the  average 
performance  rises  to  97.06%. 

The  performance  of  the  neUvork  was  then  analyzed  by  allowing  P  to  be  equal  to  the  square 
root  of  the  number  of  nodes  in  the  Kohonen  layer.  The  complete  data  are  shown  in  Tables  E.8 
and  E.9.  As  shown  in  figure  5.6  allowing  P  to  increase  as  the  number  of  nodes  increased  had  little 
effect  on  the  performance  of  the  network.  Again,  the  decrease  in  the  training  performance  for  100 


5-7 


Figure  5.6.  Performance  vs  Nodes  for  Kohonen  Training  with  Variable  P  Neighbors 


nodes  was  due  to  two  of  the  networks  converging  to  a  performance  of  less  than  85%.  If  these  two 
tests  are  removed  from  the  performance  calculations^  the  average  performance  rises  to  96.79%. 

The  maximum  robustness  of  the  network  occurred  when  the  number  of  Kohonen  nodes  was 
64  and  the  number  of  P-Neighbors  used  to  determine  the  RBF  spreads  was  eight.  This  data  is 
shown  in  Table  5.5.  Comparing  these  results  to  that  of  the  networks  trained  via  the  Nodes  at  Data 


Table  5.5.  Robustness  Measure  for  Kohonen  Training 


%  Correct 

Training 

Test 

Avg 

Std 

97^94 

1.42 

80.20 

3.84 

Points  algorithm  shows  that  the  maximum  accuracy  for  the  test  data  was  about  5%  less  for  the 
Kohonen  Training  algorithm.  Also,  when  100  nodes  were  used  to  train  the  network,  the  training 
performance  became  unpredictable.  This  is  probably  due  to  the  fact  that  the  Kohonen  Training 
algorithm  adapts  the  layer  1  weights  to  the  data  independent  of  the  class  of  the  data.  Thus,  it  is 
highly  likely  that  certain  layer  1  nodes  actually  represent  more  than  one  class  of  the  data. 

5.2,4^S  K-Means  Cluster  In  this  test,  the  ability  of  the  network  to  distribute  a  set 
number  of  nodes  to  cover  the  pattern  space,  using  the  K-Means  algorithm,  was  studied.  The 
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number  of  nodes  in  the  hidden  layer,  layer  1,  was  set  to  K,  the  number  of  clusters.  The  weights, 
or  centers,  for  the  nodes  in  layer  1  were  trained  using  the  K-Means  Clustering  algorithm  and  the 
sigmas  set  using  the  Set  Si^ma  at  P  Neighbor  Averages  algorithm.  The  weights  linking  the  nodes 
in  the  output  layer  to  the  nodes  in  the  first  layer  were  trained  via  global  minimization  of  the  MSE. 
The  performance  of  the  network  was  first  analyzed  by  setting  the  P-Neighbors  to  6  and  letting 
the  number  of  nodes  in  the  hidden  layer  vary.  The  complete  data  are  shown  in  Tables  E.IO  and 
E.‘U.  As  shown  in  figure  5.7  the  performance  of  the  network  increased  until  as  the  number  of  nodes 


Figure  5.7.  Performance  vs  Nodes  for  K-Means  Clustering  with  Six  P-Neighbors 


in  the  hidden  layer  increased.  When  number  of  nodes  reached  the  range  of  60  to  70  nodes,  the 
performance  of  the  network,  over  the  test  data,  leveled  out,  indicating. the  pattern  space  was  fully 
covered. 

The  performance  of  the  network  was  then  analyzed  by  setting  the  number  of  nodes  in  the 
hidden  layer,  layer  1,  to  60  and  varying  the  number  of  P-Neighbors  from  1  to  30.  The  complete 
da.a  are  shown  in  Tables  E.12  and  E.13.  As  shown  in  figure  5.8  the  categorization  performance 
of  the  network  was  relatively  constant  until  the  number  of  P-Neighbors  was  eight.  At  that  point, 
the  performance  of  the  network,  over  the  test  data,  decreased  slightly  as  P  was  increased.  The 
robustness  of  the  network  with  60  nodes  and  P  set  at  6  is  shown  in  Table  5.6. 

Comparing  the  performance  of  this  training  algorithm  to  the  Kohonen  Training  algorithm 
shows  the  performance  of  two  algorithms  was  roughly  equivalent.  However,  the  amount  of  time 
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Correct 


100 


Figure  5.8.  Performance  vs  P-Neighbors  for  K-Means  Clustering  with  Sixty  Clusters 


Table  5.6.  Robustness  Measure  for  K-Means  Clustering 


%  Correct 

Training 

Test 

Avg 

Std 

95.59 

1.71 

80.90 

3.36 
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required  to  train  via  the  K-Means  Clustering  algorithm  was  less  than  30  minutes  while  the  amount 
of  time  required  to  train  via  the  Kohonen  IVaining  algorithm  exceeded  120  minutes. 

Center  at  Class^Clusier  Averages  In  this  test,  the  ability  of  the  network  to 
add  the  required  number  of  nodes  to  cover  the  input  data  space,  using  the  Center  at  Class-Cluster 
Averages  algorithm,  was  measured.  The  weights,  or  centers,  for  the  nodes  in  the  first  layer  were 
trained  using  the  Center  at  Class-Cluster  Averages  algorithm  and  the  sigmas  set  using  the  Scale 
Sigmas  by  Class  Interference  algorithm.  The  weights  linking  the  nodes  in  the  output  layer  to 
the  nodes  in  the  first  layer  were  trained  via  global  minimization  of  the  MSE.  The  performance 
of  the  network  was  analyzed  by  varying  the  average  threshold  (vigilance)  of  the  nodes  in  the 
hidden  layer.  The  complete  data  are  shown  in  Tables  E.14  and  E.15.  As  shown  in  figure  5.9, 


Figure  5.9.  Nodes  vs  Average  Threshold  for  Center  at  Cl^s  Averages 


as  the  average  threshold  increased,  the  number  of  nodes  required  to  cover  the  pattern  space  of 
the  training  data  decreased.  However,  as  shown  in  figure  5.10,  as  the  number  of  nodes  decreased, 
the  categorization  performance  of  the  network  decreased.  By  comparing  both  figures,  it  can  be 
seen  that  the  categorization  performance  of  ^he  network  remains  fairly  constant  until  the  average 
threshold  increased  to  a  value  of  1.5.  At  this  point,  55  to  ^0  nodes  adequately  cover  the  pattern 
space.  As  the  average  threshold  increased  between  1.5  and”  2:5,  the  number  of  nodes  continued 
to  decrease  dramatically  while  the  classification  performance  decreased  slowly.  As  the  average 
threshold  increased  past  2.5,  the  performance  of  the  network  deteriorates  rapidly.  The  robustness 
of  networks,  trained  in  this  manner  with  an  average  threshold  of  2.0,  is  shown  in  Table  5.7. 
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Figure  5.10.  Performance  vs  Average  Threshold  for  Center  at  Class  Averages 


Table  5.7.  Robustness  Measure  of  Center  at  Class  Averages 


%  Correct 

Training 

Test 

Avg 

Std 

95.39 

1.96 

80.40 

3.17 

It  is  interesting  to  compare  the  results  of  this  training  algorithm  to  the  results  obtained  from 
the  K-Means  algorithm.  Both  algorithms  are  clustering  algorithms  which  set  the  weights  of  their 
clusters  equal  to  the  averages  of  the  features  of  pockets  of  data.  It  can  be  seen  that  the  robustness 
measure  for  the  Center  at  Class-Cluster  Averages  algorithm,  which  used  an  average  of  24  nodes  in 
the  hidden  layer,  layer  1,  was  roughly  equivalent  to  that  of  the  K-Means  algorithm  which  used  60 
nodes.  This  shows  that  centering  the  nodes  at  pockets  of  patterns  according  to  class  may  be  better 
than  centering  the  nodes  at  pockets  of  data  without  regard  to  class. 

5. ^.^.5  PNN  Training  In  this  test,  the  performance  of  the  Probabilistic  Neural  Net¬ 
work  (PNN),  developed  by  Specht,  versus  the  RBF  Kernel  Classifier  was  analyzed.  The  nurnber 
of  nodes.in  the- hidden  Jayer  was  set  equal  to  the  number  of  training  points,  102.  The  weights  for 
the  layer  1  nodes  were  set  using  the  Nodes  at  Data  Points  algorithm.  For  the  PNN,  the  output 
layer,  layer  2,  nodes  were  only  connected  to  the  hidden  layer  nodes  representing  their  class.  The 
weights  connecting  these  hidden  layer  nodes  to  the  respective  output  layer  nodes  were  set  to  one. 
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For  the  Radial  Basis  Function  (RBF)  network,  the  weights  linking  the  nodes  in  the  output  layer  to 
the  nodes  in  the  first  layer  were  trained  via  global  minimization  of  the  MSB.  The  sigmas  for  each 
network  were  then  allowed  to  vary  from  .5  to  3.0.  The  categorization  performance  of  the  network 
for  both  the  training  and  test  data  were  then  documented  as  shown  in  Tables  E.17,  E.18,  E.19,  and 
E.20  and  plotted  as  shown  in  figures  5.11  and  5.12. 


Figure  5.12.  PNN  vs  RBF  Performance  for  Test  Data 
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As  the  sigmas,  or  RBF  spreads,  for  both  networks  were  increased  to  .5,  the  performances  were 
relatively  the  same.  However,  as  the  sigma  increased  from  .5  to  1.5,  the  performance  of  the  PNN 
decreased  while  the  RBF  Network  remained  relatively  constant.  This  indicates  the  weights  in  the 
RBF  Network  are  serving  to  offset  the  choice  of  a  bad  sigma  for  the  PNN.  As  sigma  increased  from 
1.75  to  3.0,  the  performance  of  the  RBF  network  began  to  deteriorate  rapidly.  At  a  sigma  of  3.0, 
the  both  networks  performed  at  the  same  level.  The  best  performance  for  the  PNN  Network,  as 
shown  in  Table  5.8,  occurred  when  sigma  was  set  to  .5.  On  the  other  hand,  the  best  performance 


Table  5.8. 


%  Correct 

IVaining 

Avg 

Std 

Mil 

KiSisi 

ill 

N  Network 


for  the  RBF  Network,  as  shown  in  Table  5.9,  occurred  when  sigma  was  set  to  .75.  This  shows 


Table  5.9.  Robustness  Measure  of  RBF  Network 


%  Correct 

Training 

Test 

Avg 

Std 

99.61 

0.78 

83.30 

5.40 

the  increase  in  the  sigma,  or  receptive  field  spread,  allowed  the  RBF  Network  to  generalize  a  little 
better  than  the  PNN  Network. 

5.2.5  Summary  A  comparison  of  the  performance  of  the  Hyperplane  and  Kernel  Classifier 
networks  is  shown  in  Tables  5.10  and  5J1. 

These  tables  show  both  types  of  networks  performed  equally  well.  The  Hyperplane  Classifier 
networks  used  less  nodes  then  the  Ke^mel  Classifiers  but  took  longer  to  train.  Similar  observations 


Table  5.IG.  Hyperpla;ie  Classifier  Network  Robustness  Summary 


Output  Layer 
Objective  Function 

Convergence 

Iterations 

Avg  %  Correct 

Training 

Test 

Total 

MSE 

97.16 

iSi 

CE. 

100.00 

CKM 

88.83 

OSS 

81.09 
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Table  5.11 ,  Kernel  Classifier  Network  Robustness  Summary 


Layer  1 

Avg  %  Correct 

Training  Method 

Number  of  Nodes 

Training 

Test 

Total 

Node  at  Data  Points 

102 

100.00 

84.90 

Kohonen  Training 

64 

97.94 

»l 

ESI 

K-Means  Clustering 

60 

95.59 

89.20 

Center  at  Class  Avgs 

24 

95.39 

80.40 

88.85 

PNN  Network 

102 

100.00 

81.20 

90.69 

RBF  Network 

102 

99.61 

83.30 

91.54 

have  been  made  by  Moody  (13).  The  performance  of  the  different  Kernel  Classifier  networks 
depended  on  the  number  of  RBF  nodes  allocated  to  layer  1.  The  best  performance  occurred  when 
an  RBF  node  was  placed  at  each  of  the  102  training  vectors.  However,  only  slightly  degraded 
performance  occurred  when  a  lesser  number  of  nodes  was  allowed  to  adapt,  via  the  Kohonen 
Training,  K-Means  Clustering,  and  Center  at  Class-Cluster  Averages  algorithms,  to  reflect  the 
data.  Finally,  the  performance  of  the  PNN  was  inferior  to  that  of  the  RBF  based  Kernel  Classifier 
network.  This  shows  the  weights  linking  the  layer  1  nodes  to  the  layer  2  nodes  in  the  Kernel 
Classifier  may  serve  to  correct  for  non-optimal  choices  of  the  spreads  of  the  RBFs. 


5,3  Radar  Sysizm  Characierizaiion 

5.3.1  Introduction  This  section  addresses  the  development  and  testing  of  a  neural  network 
capable  of  categorizing  a  radar  platform  from  the  characteristics  of  its  electromagnetic  signal. 

5.3.2  Data  Description  The  data  deemed  necessary  to  perform  classification  of  radar  plat¬ 
forms  are  the  radio  frequency  of  the  electromagnetic  signal,  the  pulse  repetition  interval  of  the 
pulsed  waveform,  the  stagger  level  of  the  pulse  repetition  interval,  the  width  of  the  transmitted 
pulse,  the  scan  type  used  by  the  radar,  and  the  circular  period  of  the  scan. 

5.3.2.1  Radio  Frequency  The  Radio  Frequency  (RF),  as  shown  in  figure  5.13,  of  a 
radar  platform’s  transmitted  signal  represents  the  frequency,  in  hertz,  at  which  the  carrier  waveform 
is  transmitted.  This  feature  is  not  limited  to  a  specific  frequency  for  each  platform  but  can  vary 
within  a  given  a  range,  or  band,  of  frequencies  over  which  the  carrier  is  transmitted.  Also,  many 
radar  platforms  transmit  their  carrier  in  several  different  RF  bands,  depending  on  the  mode  in 
which  the  platform  is  operating. 
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Figure  5.13.  .Radar  Signal 

5.3.2.2  Pulse  Repeiiiion  Interval  The  Pulse  Repetition  Interval  (PRI),  as  shown  in 
figure  5.13,  of  a  radar  platform^s  transmitted  signal  is  the  interval  of  time,  in  microseconds,  between 
consecutive  transmitted  pulses.  This  feature  is  not  limited  to  a  specific  time  for  each  platform  but 
can  vary  within  a,given  range  of  time.  Also,  many  radar  platforrns  change  their  PRI  operating 
bands  depending  on  the  mode  in  which  the  platform  is  operating. 

5.S.2.S  Stagger  Level  The  use  of  more  than  one  pulse  repetition  frequency,  as  shown 
in  figure  5.13,  by  a  radar  platform  is  known  as  its  Stagger  Level.  This  change  in  repetition  frequency 
of  the  transmitted  pulses  allows  the  radar  to  overcome  the  blind  speeds  inherent  in  the  detection 
of  moving  targets. 

5.3.2.4  Pulse  Width  The  Pulse  Width  (PW),  as  shown  in  figure  5.13,  of  a  radar 
platform’s  transmitted  signal  is  the  time  duration,  in  microseconds,  of  the  transmitted  pulses.  This 
characteristic  determines  the  radar’s  ability  to  resolve  closely  spaced  targets  within  its  range.  This 
feature  is  not  limited  to  a  specific  unit  of  time  for  each  platform  but  can  vary  within  a  given  range 
of  times.  Furthermore,  many  rader  platforms  can  change  the  range  of  the  PW  of  their  transmitted 
signals. 

5.5.2.5  Scan  Type  The  Scan  Type  is  the  method  the  radar  platform  uses,  such  as  a 
conical  scan,  lo  direct  its  antenna  beam  at  a  target.  This  study  concentrated  o  circular  scan 
platforms  with  different  types  of  scans. 
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5. 3,2,6  Circular  Scan  Period  The  Circular  Scan  Period  is  the  length  of  time,  in  mil¬ 
liseconds,  the  radar  platform  takes  to  repeat  one  frame  of  its  search.  This  feature  is  not  limited  to 
a  specific  unit  of  time  for  each  platform  but  can  vary  within  a  given  range  of  times.  Also,  many 
radar  platforms  can  change  the  length  of  their  Circular  Scan  Period  depending  on  the  mode  in 
which  the  platform  is  operating. 

5,3,3  Data  Processing 

5, 3,3,1  Data  Normalization  The  data  used  to  develop  the  network  does  not  represent 
actual  radar  platform  features  but  sets  the  ranges  allocated  to  each  platform  in  the  form  of  a 
range  vector.  Because  the  comparative  ranges  for  each  of  the  patterns  were  unequal,  the  data 
was  normalized  by  dividing  each  feature  of  the  data  by  one-half  the  maximum  value  allowed  for 
that  feature.  This  limits  the  range  for  each  of  the  feature  components  to  a  number  between  0  and 
2.  Also,  in  this  problem,  several  of  the  ten  radar  platforms  had  features  which  overlapped  in  the 
feature  space. 


5, 3, 3, 2  Data  Generation  For  each  platform  the  training  and  test  vectors  were  gener¬ 
ated  randomly.  This  was  accomplished  by  taking  each  range  vector  within  a  class  and  generating  a 
set  number  of  random  pattern  vectors,  according  to  a  uniform  distribution,  using  the  range  vector 
as  a  template.  For  the  RF,  PRI,  PW  and  Scan  Period  features,  the  random  number  generated  was 
forced  to  reside  inside  the  ranges  allocated  to  each  platform.  For  the  Scan  Type  and  the  Stagger 
Level,  the  features  were  the  same  as  the  template  vector. 

5,3,4  Network  Development  For  this  problem,  Kernel  Classifier  networks  were  developed  to 
categorize  ten  radar  platforms.  Since  the  networks  developed  using  the  Center  at  Class-Cluster 
Averages  algorithm  performed  comparatively  well,  for  a  smaller  number  of  nodes,  for  the  commu¬ 
nications  signal  categorization  problem,  these  Kernel  Classifier  networks  were  also  developed  using 
this  algorithm.  For  these  networks,  a  correct  classification  resulted  when  the  output  of  the  correct 
classification  node  exceeded  a  classification  threshold. 

5,3,4 -1  RBP  Network  This  network  was  constructed  according  to  the  topology  shown 
in  figure  3.7.  The  input  layer,  layer  0,  contained  six  nodes  while  the  nodes  in  the  output  layer,  layer 
2,  contained- ten  nodes,  one  node  representing- each- class.  The  network  uscd=thc  Center  at'Class- 
Cluster  Averages  algorithm  to  determine  the  number  of  nodes  in  the  hidden  layer,  layer  1,  and  their 
corresponding  weights.  These  nodes  v/ere  assigned  the  gaussian  transfer  function  while  the  nodes 
in  the  output  layer  were  assigned  the  linear  transfer  function.  The  sigmas  for  the  hidden  layer 
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nodes  were  established  using  the  Scale  Sigmas  by  Class  Interference  algorithm  while  the  weights 
linking  the  output  layer  nodes  with  the  hidden  layer  nodes  set  via  global  minimization  of  the  MSB. 
Three  hundred  pattern  vectors,  30  from  each  platform,  were  then  used  to  train  the  network  while 
1990  pattern  vectors,  about  200  from  each  platform,  were  used  to  test  the  network.  Measurements 
were  then  taken  of  the  performance  of  the  network  as  the  classification  threshold  varied.  This 
data  is  shown  in  Table  E.21.  and  plotted  in  figure  5.14.  From  this  figure,  it  can  be  seen  that 


Figure  5.14.  Performance  Radial  Basis  Function  Network  for  Radar  Data 


the  categorization  performance  of  the  network  decreased  as  the  class  threshold  increased  until  a 
classification  threshold  of  A  was  reached.  At  this  point  the  categorization  performance  leveled  out 
at  93.67  %  for  the  training  data  and  82.88  %  for  the  test  data.  This  indicates  that  if  the  output 
of  a  node  was  greater  than  .4,  there  is  an  82.88%  chance  that  a  correct  classification  was  made.  If 
the  output  of  a  node  was  greater  than  .8,  there  is  a  72.29%  chance  that  a  correct  classification  was 
made.  The  inability  of  the  network  to  train  at  100  %  was  due  to  the  overlap  of  the  parameters  in 
the  feature  space  from  different  platforms. 

Arhiiraior  Network  This  network  was  constructed  according  to  the  topology 
.shown  Jn  figure  5.15.  In  this.system,  Network  A  was  trained  to  categorized  platforms  1-5  while 
Network  B  was  trained  to  categorized  platforms  6-10.  Network  C  was  trained  to  categorize  all  ten 
platforms  by  arbitrating  between  Network  A  and  Network  B. 

For  Networks  A  and  B,  the  input  layer,  layer  0,  contained  six  nodes  while  the  nodes  in 
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Figure  5.15.  Radar  Data  Arbitration  Network 


their  output  layer,  layer  2,  contained  five  nodes,  one  node  representing  each  class.  These  networks 
used  the  Center  at  Class-Cluster  Averages  algorithm  to  determine  the  number  of  nodes  in  the 
hidden  layer,  layer  1,  and  their  corresponding  weights.  These  nodes  were  assigned  the  gaussian 
transfer  function  while  the  nodes  in  the  output  layer  were  assigned  the  linear  transfer  function.  For 
Network  A,  the  sigmas  for  the  hidden  layer  nodes  were  established  using  the  Scale  Sigmas  by  Class 
Interference  algorithm  while,  for  Network  B,  the  sigmas  were  set  using  the  P-Neighbors  algorithm. 
For  both  networks,  the  weights  linking  the  output  layer  nodes  with  the  hidden  layer  nodes  were 
set  via  global  minimization  of  the  MSE.  Three  hundred  pattern  vectors,  60  from  each  platform, 
were  then  usi.d  to  train  each  network  independently.  Network  A  was  then  tested  with  1000  pattern 
vectors  from  classes  one  through  five,  group  A,  while  Network  B  was  tested  with  990  pattern  vectors 
from  classes  six  through  ten,  group  B.  Measurements  were  taken  of  the  performance  each  network 
for  as  the  classification  threshold  varied.  This  data  is  shown  in  Tables  E.22  and  E.23  and  plotted 
in  figures  5.16  and  5.17. 

After  the  parameters  for  Network  A  and  B  were  set,  Network  C  was  established  to  arbitrate 
the  outputs  between  Networks  A  and  B.  This  network  had  ten  nodes  in  its  input  layer,  one  for 
each  of  the  ten  outputs  from  the  Networks  A  and  B,  and  ten  nodes  in  its  output  layer;  one  for 
each  radar  platform.  The  Center  as  Class-Cluster  Averages  algorithm  was  used  to  determine  the 
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Figure  5.17.  Performance  of  Network  B  for  Group  B  Radar  Data 
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number  of  nodes  in  its  hidden  layer,  layer  1,  and  their  corresponding  weights.  These  nodes  were 
assigned  the  gaussian  transfer  function  while  the  nodes  in  the  output  layer  were  assigned  the  linear 
transfer  function.  The  sigmas  for  the  hidden  layer  nodes  were  established  using  the  Scale  Sigmas 
by  Class  Interference  algorithm  while  the  weights  linking  the  output  layer  nodes  with  the  hidden 
layer  nodes  set  via  global  minimization  of  the  MSE.  Three  hundred  pattern  vectors,  30  from  each 
platform,  were  then  used  to  train  the  network  while  1990  pattern  vectors,  about  200  from  each 
platform,  were  used  to  test  the  network.  Measurements  were  then  taken  of  the  performance  of  the 
network  as  the  classification  threshold  varied.  This  data  is  shown  in  Table  E.24  and  plotted  in 
figure  5.18. 


Figure  5.18.  Performance  of  Arbitrator  Network  for  Group  A  and  B  Radar  Data 


From  these  figures,  it  can  be  seen  that  the  categorization  performance  of  the  network  increased 
as  the  class  threshold  decreased  until  the  class  threshold  reached  .4  .  At  this  point  the  classification 
accuracy  of  the  total  network  was  99.33%  for  the  training  data  86.35%  for  the  test  data.  This 
indicates  that  if  the  output  of  a  node  was  greater  than  .2,  there  is  an  86.35%  chance  that  a  correct 
classification  was  made.  If  the  output  of  a  node  was  greater  than  .8,  there  is  a  73.90%  chance  that 
a  correct  classification  was  made.  The  inability  of  the  network  to  train  at  100  %  was  due  to  the 
overlap  of  the  parameters  in  the  feature  space  from  different  platforms. 

5.34»S  Summary  A  summary  of  the  performance  of  the  networks  trained  to  classify 
the  radar  data  is  shown  in  Table  5.12.  A  comparison  of  the  performance  of  the  RBF  network  and 
the  Arbitrator  network  shows  the  Arbitrator  network’s  performance  was  about  2%  better  than  that 
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Table  5.12,  Radar  Categorization  Summary 


Classification 

Threshold 

RBF  Network  %  Correct 

Arbitrator  Network  %  Correct  _ 

Training 

Test 

Training 

Test  II 

93.67 

99.33 

86.35 

.0 

93.67 

82.88 

99.33 

86.35 

of  the  RBF  network.  This  is  probably  due  to  the  ability  to  the  use  of  more  training  vectors  to  train 
networks  A  and  B  in  the  Arbitrator  scheme. 


5,4  Summary 

This  chapter  began  by  discussing  the  data  used  to  train  and  test  neural  networks  to  classify 
coded  digital  communication  signals.  After  describing  the  methods  used  to  train  various  networks 
to  solve  this  classification  problem,  the  results  of  the  training  and  testing  were  then  presented.  This 
chapter  concluded  with  a  discussion  of  the  data  used  to  train  and  test  neural  "networks  to  classify 
radar  systems  and  an  analysis  of  these  training  and  test  results. 
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VL  Conclusions/Recommendations 


6.1  Introduction 

The  purpose  of  this  thesis  was  to  characterize  the  Hyperplane  and  Kernel  Classifier  types  of 
neural  networks  and  determine  either  type  could  be  used  to  accurately  characterize  radar  signal. 
After  drawing  some  conclusions  on  the  performance  of  each  type  of  network,  based  the  test  results 
discussed  in  Chapter  5,  this  chapter  will  recommend  areas  of  future  study.  ^ 

6.2  Conclusions 

6.2.1  Hyperplane  Classifier  In  this  thesis,  Hyperplane  Classifier  networks  were  constructed 
using  the  MSB,  CE  and  CFM  objective  functions.  For  the  communications  problem,  each  network 
contained  18  nodes  in  the  first  hidden  layer,  ten  nodes  in  the  second  hidden  layer  and  two  nodes 
in  the  output  layer.  Under  these  conditions,  the  network  trained  by  minimizing  the  CE  performed 
slightly  better,  in  the  area  of  classification,  than  the  networks  trained  via  MSE  and  CFM  objective 
functions.  The  average  CE  performance  on  the  training  data  was  100%  while  on  the  test  data  the 
performance  dropped  to  81.2%.  This  compares  favorably  with  the  MSE  performances  of  97,16%  and 
79.7%  and  the  CFM  performances  of  88.83%  and  73.20%  for  the  training  and  test  data  respectively. 
Furthermore,  the  CE  algorithm  converged  about  twice  as  fast  as  the  MSE  algorithm  and  seven  times 
as  fast  as  the  CFM  algorithm.  As  shown  in  figures  5.1,  5.2,  and  5.3,  the  CE  algorithm  reached  a 
90%  accuracy  in  about  7000  iterations  while  the  MSE  and  CFM  algorithms  reached  this  level  in 
about  19000  and  50000  iterations  respectively.  This  decrease  in  convergence  time  is  due  to  the  lack 
of  a  yn{l  -  Vn)  term  in  the  CE  update  equations. 

6.2.2  Kernel  Classifier  For  this  thesis,  several  different  Kernel  Classifier  networks  were 
developed  to  solve  the  communications  signal  categorization  problem  discussed  in  Chapter  5.  Based 
on  the  performance  results  detailed  in  Chapter  5,  two  Kernel  Classifier  types  of  networks  were  then 
constructed  to  categorize  radar  systems. 

6. 2.2.1  Communications  Data  The  performance  of  networks  trained  using  the  Nodes 
at  Data  Points  was  found  to  be  a  critical  function  of  the  number  of  nodes  used  in  the  hidden  layer. 
As  the  number  of  nodes  increased  from  0  to  102,  the  classification  performance  of  the  network 
increased  accordingly;  peaking  at  100%  for  the  training  data  and  84.90%  for  the  test  data  for  102 
nodes.  However,  for  less  than  60  nodes,  the  performance  of  the  network  deteriorated  rapidly. 

The  performance  of  the  networks  trained  using  the  Kohonen  Training  algorithm  were  found 
to  be  a  function  of  the  both  the  number  of  nodes  allocated  to  the  hidden  layer  and  the  number  of 
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P  nearest  neighbors  used  in  the  calculation  of  the  spreads  of  the  RBFs.  As  the  number  of  nodes 
increased  from  0  to  81  the  classification  performance  of  the  network  increased,  peaking  at  100%  for 
the  training  data  but  only  around  76%  for  the  test  data.  This  is  due  to  the  fact  that  the  Kohonen 
Training  algorithm  allocates  more  nodes  to  areas  in  the  feature  space  where  there  are  pockets  of 
data.  Increasing  the  total  number  of  nodes  in  the  hidden  layer  increases  the  amount  of  nodes 
which  can  be  allocated  to  each  pocket.  However,  this  increase  in  the  number  of  nodes  within  these 
pockets  serves  to  decrease  the  distance  to  the  P  nearest  neighbors.  Thus,  the  spreads  for  the  RBFs 
become  small  and  the  network  loses  its  ability  to  generalize  past  the  training  data.  As  shown  in 
figure  5.6,  this  problem  can  be  overcome  by  using  less  nodes  in  the  Kohonen  layer.  For  this  data, 
the  optimum  number  of  nodes  was  64.  Though  the  performance  over  the  training  data  was  only 
97.94%,  the  performance  over  the  test  data  improved  to  80.20%. 

The  performance  of  the  networks  trained  using  the  K-means  algorithm  were  also  found  to  be 
a  function  of  the  number  of  nodes  allocated  to  the  hidden  layer.  As  the  number  of  nodes  increased 
from  0  to  100,  the  classification  performance  of  the  network  generally  increased;  peaking  at  100% 
for  the  training  data  and  83.90%  for  the  test  data.  The  performance  remained  above  80%  for  both 
the  test  and  training  data  until  less  than  60  nodes  were  allocated  to  layer  1.  At  this  point  the 
classification  performance  of  the  network  decreeised  rapidly.  This  is  probably  due  to  the  grouping 
of  dissimilar  classes  into  the  same  clusters.  The  performance  of  networks  trained  using  the  K-means 
algorithm  were  found  to  be  somewhat  invariant  to  the  number  of  P  nearest  neighbors,  used  in  the 
calculation  of  the  spreads  of  the  RBFs. 

The  performance  of  networks  drained  using  the  Center  at  Class- Cluster  Averages  algorithm 
was  also  found  to  be  A  function  of  the  number  of  nodes  r.sed  in  the  uidden  layer.  As  the  number  of 
nodes  increased  from  0  to  100,  the  classification  performance  of  the  network  increased  accordingly, 
peaking  at  100%  for  the  training  data  and  84%  for  the  test  data.  This  indicates  there  was  redundant 
lata  in  the  training  set.  Also,  networks  trained- in  this  manner  performed  at  a,  performance  level 
of  aboY^  80%,  for  botL  the  treming  and  test  data,  as  long  as  tl(e  number  of  nodes  remained  above 
20.  This  indicates  the  data  in  this  problem  may  be  grouped,  by  cl?ss,  into  small  pockets  of  data. 

The  performance  of  the  networksitrained  usin^  ihe  PNN  algorithm  were  found  to  be  a  function 
of  the  spreads  assigned  to  the  RBF  node.5.  As  the  spreads  decreased,  from  3  to  .25,  the  classification 
performance  of  the  network  increased;  peaKing  at  100%  fcj  the  training  data  and  81.29%  for  the 
test  data.  The  performance  of  an  RBF  netw^^-k,  with  the  weii^lits  for  the  hidden  layer  nodes  trained 
in  the  same  manner  at  those  for  the  PNN,  was  found  to  be  less  of  a  function  of  the  spread  assigiied 
to  the  RBF  nodes.  As  shown  in  figures  5.11  and  5.12;  ever?  as  the  spreads  increased  to  1.5,  the 
performance  of  the  RP^  network  remained  relatively’ constant  for  both  the  training  and  test  data. 
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This  is  due  to  the  weights  in  the  output  layer  compensating  for  the  poor  choice  of  the  spread  in 
the  hidden  layer.  However,  as  the  spread  of  the  RBFs  increased  past  1.5,  the  performance  of  the 
RBF  network  deteriorated  to  that  of  the  PNN. 

6,k.2,2  Radar  Data  For  this  problem,  ten  different  radar  platforms  had  to  be  catego¬ 
rized  from  data  concerning  their  electromagnetic  signals. 

The  performance  of  the  standard  RBF  network  developed  to  solve  this  problem  was  a  function 
of  the  classification  threshold  required  to  assign  a  correct  decision  for  the  network.  For  a  classi¬ 
fication  threshold  of  .8,  indicating  the  output  for  the  correct  node  was  above  .8,  the  performance 
of  the  network  was  87%  for  the  training  data  and  72.29%  for  the  test  data.  As  the  classification 
threshold  was  decreased  to  .4,  the  performance  of  the  network  increased  to  93.67  %  and  82.88  %  for 
the  training  and  test  data  respectively.  This  performance  then  remained  constant  as  the  threshold 
decreased.  The  inability  of  the  network  to  train  at  the  100%  level  indicated  the  overlap  in  the 
feature  space  between  patterns  of  different  classes.  The  10%  difference  between  the  performance  of 
the  network  for  training  and  test  data  followed  the  general  performance  degradation  found  in  the 
communications  problem. 

The  performance  of  the  Arbitrator  network  developed  to  solve  this  problem  was  also  a  function 
of  the  classification  threshold  required  to  assign  a  correct  decision  for  the  network.  For  a  classi¬ 
fication  threshold  of  .8,  indicating  the  output  for  the  correct  node  was  above  .8,  the  performance 
of  the  network  was  95%  for  the  training  data  and  73.90%  for  the  test  data.  As  the  classification 
threshold  was  decreased  to  .2,  the  performance  of  the  network  increased  to  99.33  %  and  86.35  %  for 
the  training  and  test  data  respectively.  This  performance  then  remained  constant  as  the  threshold 
decreased.  Again,  the  failure  of  the  network  to  train  a  100  %  indicates  the  overlap  in  the  features 
space  between  patterns  of  different  classes. 

In  general,  the  performance  of  the  Arbitrator  network  was  between  1.6%  to  3.5%  better 
than  that  of  the  standard  RBF  network.  This  is  due  to  the  establishment  of  the  two  subnetworks 
to  distinguish  between  smaller  groups  of  platforms,  the  use  bf  more  training  data  to  train  these 
networks,  and  the  of  ability  of  the  Arbitrator  network  to  separate  the  overlap  between  the  groups. 
However,  this  increase  in  performance  for  the  Arbitrator  network  required  almost  three  times  as 
many  nodes  as  that  of  the  standard  RBF  network.  Furthermore,  since  the  two  subnetworks  were 
trained  separately,  the  total  training  time  for  tbe  Arbitrator  network  was  almost  three  times  that 
of  standard  RBF  network  of  45  minutes. 
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6,S  Recommendaiions 


IVom  the  results  found  in  this  thesis,  better  algorithms  need  to  be  developed  to  set  the 
spreads  of  the  radial  basis  function  nodes  for  the  Kernel  Classifier  RBF  networks.  In  order  to 
optimize  the  performance  of  the  networks,  a  trail  and  error  basis  was  used  to  select  the  algorithm, 
and  algorithm  parameters,  to  set  the  spreads.  A  single,  adaptive,  algorithm  to  accomplish  this 
same  task  could  improve  the  performance  of  these  networks.  Also,  it  may  be  beneficial  to  study 
accuracies  of  Kernel  Classifier  networks  developed  using  the  backpropagation  algorithms  to  set  the 
weights  connecting  nodes  in  layer  1  to  nodes  in  layer  2.  In  this  this  thesis,  these  weights  were  set  via 
a  global  minimization  of  the  MSE  using  a  matrix  inversion  algorithm.  Since  this  algorithm  used 
Gaussian  Elimination  of  the  rows  to  obtain  the  inverse,  it  is  possible  that  small  roundoff  errors 
could  have  accumulated  in  such  a  manner  as  to  prevent  a  true  global  MSE  minimization  from 
being  obtained.  Another  area  of  research,  which  may  hold  promise,  is  the  use  of  the  RBF  networks 
to  reconstruct  functions  from  past  samples.  These  networks  could  be  useful  in  signal  processing 
applications  in  which  a  reconstruction  of  an  unknown  signal  is  needed  in  a  timely  manner. 

6.4  Summary 

This  chapter  presented  some  conclusions,  based  on  the  test  results  discussed  in  Chapter  5,  on 
the  performance  of  the  Hyperplane  and  Kernel  Classifier  neural  networks.  For  the  data  processed 
in  this  theses,  it  was  found  that  the  Kernel  Classifier  networks  could  perform  at  the  same  level 
as  that  of  the  Hyperplane  Classifiers  and  be  developed  in  a  much  shorter  time  period.  However, 
these  Kernel  Classifier  networks  usually  required  more  nodes.  Some  areas  of  future  research  may 
include  the  development  of  algorithms  to  automatically  establish  the  spreads  of  the  RBFs  in  the 
Kernel  Classifiers  and  the  application  of  Kernel  Classifer  networks  to  the  problem  of  function 
reconstruction  based  on  sampled  values. 
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Appendix  A.  Objective  Function  Analysis 


AJ  Iniroduciion 

In  this  appendix,  the  classification  properties  of  the  Mean  Square  Error,  Cross  Entropy  and 
Classification  Figure  of  Merit  objective  functions  will  be  analyzed. 


A,2  Mean  Square  Error  (MSE)  Function 

This  section  will  show  how  the  mean  square  error  for  proper  network  classification  can  be 
greater  than  the  mean  square  error  for  an  improper  classification. 

Suppose  the  network  was  required  to  make  a  classification  of  an  unknown  pattern  into  one  of 
N  classes.  That  is,  the  neural  network  was  developed  such  that  each  output  node  represents  only 
one  of  the  N  classes.  Let  dn  be  the  desired  output  for  the  node  for  a  given  input  pattern.  The 
MSE,  for  a  given  input  pattern,  is  then  defined  as 

1  ^ 

MSE=-^Yl(yn-<^nf  (A.l) 

n=l 

Usually  during  training,  dn  is  taken  to  be  1  for  the  node  responsible  for  a  class  and  0  for  the  rest  of 
the  output  nodes.  Specifically,  assume  that,  after  training,  a  test  pattern  is  applied  and  the  correct 
node  on  the  output  layer  has  the  highest  activation.  In  this  case  the  msoiimum  MSE  is 

MSE,„a^  Ri  (A.2) 

This  occurs  when  the  desired  node  produces  a  1  and  all  of  the  other  nodes  produce  activation  values 
very  close  to  1.  That  is 


MSEma.  =  ^  "  dnf  =  -{(1  -  1)' +  "  «)'! 


n=l 


(A.3) 


n=:l 


This  reduces  to 


(A.4) 


n=l 


Simplifying, 
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MSErna^  =  (A.5) 

T^ext,  assume  that,  after  training,  a  test  pattern  is  applied  and  an  incorrect  classification  is 
made.  That  is,  a  node  representing  an  incorrect  classification  in  the  output  layer  has  the  highest 
activation.  In  this  case  the  minimum  MSB  is 

MSEmin  =  ^  (A;6) 

This  occurs  when  the  output  for  the  correct  node  is  approximately  .5  and  the  output  for  a  single 
incorrect  node  is  approximately  .5.  The  outputs  for  the  rest  of  the  nodes  are  zero.  That  is 

=  (A.7) 

n=ri‘ 

Substituting  the  output  values  gives 

MSEmin  =  ~[il  ~  +  (3  -  .5)2]  (A.8) 

which  simplifies  to 


MSE„,i„  =  ^  (A.9) 

Thus,  under  certain  conditions,  the  MSE  for  a  correct  njsponse,  in  which  the  correct  output 
node  is  the  highest,  con  be  greater  than  the  MSB  for  an  incorrect  response  As  the  number  of 
dasses  becomes  very  largo,  , then  A/Sj^moa:  for  a  correct  wponse  approaches  'I  while  MSEmin  for 
an  incorrect  response  apprOt;ch<«  p.  Therefore,  Uie  ratio  of  MSSm.x  to  MSE^itn  is 


hmff^co  j^fSEmin  -  0  " 


(A.10) 


A.^  1  Cross  Enifupy  (iGE)  Function 

This  sectioii'Wiil  show  how  the.  cross  entropy  for  proper  network  classification  can  be  greater 
theh'thc'  cross.entropy  idr  an  impreprt  classification. 


A-2 


Suppose  the  network  was  required  to  make  a  classification  of  an  unknown  pattern  into  one  of 
N  classes.  That  is,  the  neural  network  was  developed  such  that  each  output  node  represents  only 
one  of  the  N  classes.  Let  dn  be  the  desired  output  for  the  node  for  a  given  input  pattern.  The 
CE  is  then  defined  as 


^(4  log(y„)  +  (1  -  4)  log(l  -  yn)]  (A.il) 

n=l 

Here  during  training,  dn  is  usually  set  to  1  for  the  node  responsible  for  a  class  and  to  0  for  the  rest 
of  the  output  nodes.  Now,  assume  that,  after  training,  a  te^t  pattern  is  applied  and  the  correct 
node  on  the  output  layer  has  the  highest  activation.  In  this  the  maximum  GEJs 


^  ^  (A.12) 

This  occurs  when  the  correct  classification  node  produces  a  1  and  at  least  one  of’ the  other  nodes 
produces  an  activation  value  very  close  to  1.  TJiat  is 

I  N 
nrl 

Substituting  the  values  for  the  outpuli# 

GE,„a.  =  -;^[1  log(l)  +  (1)  log(l  -  1)]  (A.14) 


or 


GEmax  —  oo  (A.1.5) 

Next,  assume  that,  after  training,  a  teat  pattern  is  applied  and  an  incorrect  node  on  the 
output  layer  has  the  highest  activation.  In  this  case  the  minimum  CE  is 


CEmin  = 


2log(2) 

N 


(A.16) 


This  occurs  when  the  output  for  the  correct  node  is  approximately  .5  and  the  output  for  a  single 
incorrect  node  is  approximately  .5.  The  outputs  for  the  test  of  the  nodes  are  «erp.  That  is 
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(A.17) 


CE  =  —j^  log(y„)  +  (1  -  d„)  log(l  -  y„)) 
n=:l 

Substituting  the  assumed  values  for  the  outputs 

=  -  ;^(1  M-5)  +  (1  -  0)  log(l  -  .5)]  (A.18) 

which  simplifies  to 


CEmin  — 


21og(2) 

N 


(A.19) 


Thus,  under  certain  conditions,  the  CE  for  a  correct  response,  in  which  the  correct  output 
node  is  the  highest,  can  be  greater  than  the  CE  for  an  incorrect  response.  The  ratio  of  CEmax  for 
a  correct  response  to  CEmin  for  an  incorrect  response  is 


limisf. 


CE„ 


■'max 

*00  —  OO 


CEn 


(A.20) 


A.4  Classificaiion  Figure  of  Merit  (CFM)  Function 

This  section  will  show  how  the  CFM  objective  function  seeks  to  alleviate  the  classification 
error  associated  with  the  MSE  and  CE  objective  functions. 

Suppose  the  network  was  required  to  make  a  classification  of  an  unknown  pattern  into  one  of 
N  classes.  That  is,  the  neural  network  was  developed  such  that  each  output  node  represents  only 
one  of  the  N  classes.  The  CFM  objective  function  is  then  defined  as  follows: 


CFM= - —  - - - 

n=ln5tc 

where  =  2/c  “  Vn 

t/c  =  response  of  the  correct  node 
=-response  otthedncorrect  node 
N  =  total  number  of  output  nodes  or  classes, 
or  =  sigmoid  scaling  parameter. 

/?  =  sigmoid  discontinuity  parameter. 


(A.21) 
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C  =  sigmoid  lateral  shift  parameter. 

Specifically,  assume  that,  after  training,  a  test  pattern  is  applied  to  the  network  and  the 
correct  node  on  the  output  layer  has  the  highest  activation.  In  this  case  the  minimum  CFM  is 

CFMmin  «  Cf'Mn(O)  (A.22) 


where 


CWn«„)=  (A.23) 

This  occurs  when  the  correct  output  node  produces  a  1  and  all  of  the  other  output  nodes  produce 
activation  values  very  close  to  1.  That  is 


CFM 


1  ^ 

_  1  oc 

^  N-  1  ^ 


n=l„^c 

but  here  Sn  —  ye  — Vn^O  and  therefore,  CFMmin  simplifies  to 


H) 


(A.24) 


CFM„ 


_E  =  E  C™(0) 


(A.25) 


which  can  be  written  as 


CFM„,in  »  -  l)CFM(O) 


(A.26) 


This  simplifies  to 


CFM,r.i„  =  CFM{0) 


(A.27) 


Next,  assume  that,  after  training,  a  test  pattern  is  applied  and  an  incorrect  node  on  the 
output  layer  has  the  highest  activation.  In  this  case  the  maximum  CFM  is 


CFM,r,o^  =  -  2)CFM„(1)  +  CFM„(0)] 


(A.28) 
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This  occurs  when  the  output  for  the  correct  node  is  approximately  1  and  the  output  for  one  of  the 
remaining  nodes  is  approximately  1  while  the  rest  are  zero.  For  the  nodes  which  have  output  very 
near  zero 


CFMiS,)  =  ,  ^  =  CFMil)  (A.29) 

The  equation  for  the  CFM  can  now  be  written  as 

«  =  ln^c 

This  can  be  written  as 


1 

CFM„,a^  =  j^[Y,(^FMW  +  <^FMiO)]  (A.31) 

Expanding  via  the  summation 

=  -^[{N  -  2)CFM{1)  +  C7iJ’i\/(0)]  (A.32) 

Now,  taking  the  limit  as  N  approaches  infinity 

GFMmax  «  CFMn{l)  (A.33) 

For  large  /?,  then  GFMmax  ^  1-  Thus,  the  ratio  of  GFMmax  for  a  correct  response  to  CFMmin 
for  an  incorrect  response,  when  =  0  is 


CFM„ 


GFM„ 


=  2 


(A.34) 


Comparing  this  to  the  CE  and  MSE  functions,  we  find  that  the  CFM  function  has  less  region 
in  the  feature  space  where  an  incorrect  classification  will  be  made. 
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Appendix  B,  Hyperplane  Classifier  Parameter  Update  Equations 

B.l  Iniroduciion 

In  this  appendix,  the  update  equations  for  each  of  the  objective  functions  will  be  derived. 
These  equations  will  be  based  on  a  network  with  the  topology  defined  in  figure  3.3. 

B.2  Identities 

In  this  section,  the  identities  needed  to  establish  the  update  algorithms  will  be  derived.  Con¬ 
sider  a  feedforword  artificial  neural  network  as  shown  in  figure  3  with  the  following  paramenters: 

Layer  0,  input  layer,  has  K  possible  nodes 

Layer  1,  first  hidden  layer,  has  L  possible  nodes 

Layer  2,  the  second  hidden  layer  has  M  possible  nodes. 

Layer  3, the  output  layer,  has  N  possible  nodes. 

Let  the  weights  between  layers  be  defined  as  follows: 

Wki  =  weight  linking  node  k  in  layer  0  to  node  1  in  layer  1. 

wim  =  weight  linking  node  1  in  layer  1  to  node  m  in  layer  2. 

Wmn  =  weight  linking  node  m  in  layer  2  to  node  ndn  layer  3. 

Let  the  offsets  for  the  nodes  in  each  layer  be  defined  as  follows: 

ai  =  offset  for  node  1  in  layer  1. 

(Ttn  =  offset  for  node  m  in  layer  2. 

an  =  offset  for  node  n  in  layer  3. 

Let  the  transfer  function  for  the  nodes  in  each  layer  be  as  follows: 

Layer  0  -  2/jk  =  xjb  Outputs  are  same  as  input  features. 

Layer  1  -  y/  =  [1  + 

Layer  2  -  =  (1  + 

Layer.  3  -  y„  =.(1  +  e"Cm=i  “'m«ym+<rn)j-i 

Now,  looking  at  the  node  outputs  for  each  of  the  layers,  the  following  identities  need  to  be 
established: 

QVN  dVM  dVM  dyi 
>  OwkL  * 
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starting  with  y//,  the  output  for  a  node  in  layer  3 


,  =  _ £_[1  + 

dtOMN  dWMN  ^ 


(B.l) 


which  expands  to 


dyN 

dWMN 

which  simplifies  to 


-[1  +  e“(Sm=i  ^[e-(Em=,  "'mwymT<Tw)i  (3  2) 

dWMN 


dVN 

dWMN 


M 


=  _[l  +  e-(E.=.“'™'^Vm+^N)]-2[_e-(E„=.  Wmwym  +  <t7v)  (B.3) 

m=:l 


This  equation  can  be  written  as 


dyN 

dWMN 


;  [l+e-<Em=,  (B.4) 


which  finally  simplifies  to 


dyN 


dWMN 

Similarly,  using  the  same  arguments  as  above 


=  yiv(i  —  yN)yM 


(B.5) 


and 


Now  let’s  find 


dyM 

dwiM 


dyi 

dwKL 


=  yAf(i  -  yM)yL 


=  !/i(i  -  yL)yK 


(B.6) 


(B.7) 


■  =  _ - _ (1  4-  g-^mAr^-l 

dwi^M  dWLM  ^ 


(B.8) 
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where 


M 

<l>mN  =  ^mNym  + 
mrrl 


therefore 


dWLM  ~ 


which  expands  to 


dpN 

dwiM 


this,  in  turn,  simplifies  to 


dpN 

dWLM 


(1  + 


d 

dWLM 


i4mN) 


which  can  be  written  as 


which  simplifies  to 


dVN 

dWLM 


yAf(l-yAr) 


d 

dWLM 


(E 


m=l 


WmNym  +  <^n) 


dyN 

dwiM 


=  yw(i  -  yN)wMN 


dyjg 

divLM 


Substituting  equation  B.6  for  gives 


dyN 

dwiM 


=  2/Ar(l  -  yN)wMNyM(}  -  yM)yL 


Similarly 


dym 

dWKL 


=  yM(i  -  yM)wLMyL(X  -  yL)yK 


Finally,  let’s  find 


* 


dWKL  dWKL 


(B.9) 


(B.IO) 


(B.ll) 


(B.12) 


(B.13) 


(B.14) 


(B.15) 


(B.16) 


(B.17) 
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where 


M 


4>Nm  =  WmNVm  +  <^N 


m=l 


therefore 


dytf  d 


dWKL  dWKL 


which  expands  to 


dyN 

dWKL 


which  further  expands  to 


Af 


^  =  (1  +  +  n) 

m=l 


this  simplifies  to 


(B.18) 


(B.19) 


(B.20) 


(B.21) 


dyN  oym 

Now  substituting  from  equation  B.16  for  provides 


M 


dyn 


(B.22) 


dyN 

dWKL 


M 


=  yAf(l  -  yN)  ^  «'m/vym(l  -  ym)wimyLiX  -  yL)yK  (B.23) 


m=l 


Finally,  let’s  find  |^,  and  Starting  with  j/aT)  the  output  for  a  node  in  layer  3 


which  expands  to 


dyN 

dan 


-  _£_[i  +  e"(Sm=i 
da^  dcrpj  ^ 


_(1  +  e"(I^m=l  W„NVm^ON) 

dffN 


(B.24) 


(B.25) 


which  simplifies  to 


B-4 


=  _[l  +  (B.26) 


This  equation  can  be  written  as 


=  [1  +  e"(2m=l  “'">''Vm+<7^,)J-lJ^  ^  e-(Ell!=l  «"mNym+<Tw)j  (B_27) 

which  simplifies  to 


djjN 

d(TN 


=  2/Ar(l  -  dn) 


(B.28) 


Similarly,  using  the  same  arguments  as  above 


OyM 

d(7M 


=  J/m(1  -  Dm) 


(B;29) 


and 


(B.30) 


Now  let^s  find 

OQM 


dcTM  d<TM 


(B.31) 


where 


M 


4>mN  =  '^mNym  +  (^N 


m=l 


(B.32) 


therefore 


9<rM 


(B.33) 


which  expands  to 
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(B.34) 


dpN 

d(TM 


d 


(g-^mW) 


this,  in  turn,  simplifies  to 


which  can  be  written  as 


dyN 

dcTM 


WmArym  +  <^n) 


which  simplifies  to 


dyN 

d<TM 


=  yAr(l  -  yjv)«'WAf 


dyM 

dCTM 


Substituting  from  equation  B.29  for  provides 


=  yA?(i  -  yAr)«'MAryj»f(l  -  yw) 

Similarly 


dyM 

dai 


=  yM(i  -  yitf  )w£,wy£(i  -  yi) 


Finally,  lets  find 


dyN 

dcTL 


d 

dffi 


where 

M 

^Nm  —  ^  ^mNym  4"  ^N 

m=l 

therefore 


dyN  _ 

dcTL  ~  d<Tl 


1 


(B.35) 


(B.36) 


(B.37) 


(B.38) 


(B.39) 


(B.40) 


(B.41) 


(B.42) 
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which  expands  to 


dyn 

dai 


-(1  + 


(B.43) 


which  further  expands  to 


%Ar 


=  (1  +  Wn^NVm  +  vn) 

m—1 


M 


(B.44) 


this  simplifies  to 


^  =  yjv(l  -  yAf)(X] 

m=l 

and  thus,  substituting  equation  B.39  for  provides 

=  2/iv(l  -  y/v)  ^m/srym(l  -  ym)^LmyL{l  ~  yi)  (B.46) 

m=:l 

With  these  identities,  the  incremental  update  equations  for  networks  implemented  using  the 
MSB,  CE  and  CFM  objective  functions  can  be  found. 


B»S  Mean  Square  Error  (MSE) 

The  inCi'emental  update  rules  for  the  network  parameters  can  be  found  by  minimizing  the 
MSE  function  with  respect  to  the  network  parameters  for  an  instantaneous  pattern.  The  MSE 
function  is  defined  as 


AT 


(B.47) 


n=l 


The  update  equations  for  the  parameters  will  be 


^MN  =  ^MN  -  n 


dMSE 

dwMN 


(B.48) 


and 
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^LM  =  «'Zm  -  n 


dMSE 

dwtM 


(B.49) 


^KL  =  ^KL  -  »?• 


dMSE 
^  dWKL 


(B.50) 


^N-<^N-  V- 


dMSE 


(B.51) 


dMSE 


(B.52) 


dMSE 


(B.53) 


Here  rj  is  constant  which  determines  how  much  each  parameter  is  updated  for  a  given  iteration. 
Taking  the  derivative  with  respect  to  the  output  layer  weights 


dhdSBj  d  f  1  I  v<»2 

dWMN  ~  dWMN  N  ^  " 


(B.54) 


This  simplifies  to 


dMSE  2  .  ,  dytf 

dxDMN  N  ^  dwMN 


(B.55) 


Substituting  B.5  for  provides 


=  ^{VN  -  dN)yNil  -  yN)yM 


(B.86) 


Now  let’s  find  • 


B-8 


dMSE  _  d  ,l  ^ 

dWLM 


(3.57) 


Simplifying 


dMSE  ^  2  ,  ,  5y„ 

dwtM  ~ 


(B.58) 


Substituting  equation  B.15  for  provides 


=  jf  “  ‘^n)yn(l  -  yn)WMnyM{l  “  yM)yL 

4  =  1 


(B.59) 


Finally,  let’s  find 


>o  fin/4  dMSE 


dMSE  d  ,  1 


This  simplifies  to 


jmo£/  u  ,  1  .,2 

dWKL  ~  dWKL^N]^}^" 


dMSE  _  2  A  ay„ 

dWKL  ^ 


Substituting  B.23  for  -/j"-  provides 


(B.60) 


(B.61) 


dMSE  2 


- =  Jf  Y^iyn  -  d„)y„{l  -  y„)[53  Wmn(ym(l  “  ym)t«imj/4i(l  “  yL)yK]  (B.62) 

n=l  m=l 


Now  lets  find  the 


dMSE  5  , 1  A/  , 

n=l 


(B.63) 


This  can  be  written  as 


dMSE  2,  ,  .dytf 


(B.64) 


Substituting  B.28  for.  provides 


dMSE  2.  .  -  , 

•  g— -  ■  =  jy(j/Ar  -  dAr)y/v(l  -  Vn) 


(B.65) 


Now  lets  find  the 


N 


dMSE  d  - 1  v^/  I  m2 


(B.66) 


This  can  be  written  as 


dMSE  2 


N 


UViOJl/  L  j 


(B.67) 


Substituting  B.38  for  provides 


dMSE  2 


if 


Q---  =  YliVn  -  dn)ynO-  “  ynjWMnyAfi^  -  yin)  (B.68) 


Now  lets  find  the 


N 


dMSE  d  ,1  j  v-,2 

n=l 


(B.69) 


This  can  be  written  as 


dMSE  2  ,  ,  ay„ 

n=l 


(B.70) 


Substituting  for  B.46  for  provides 


^  -  dn)yn{l  -  yn)[X^  t«mnym(l  “  ym)wLmyLil  “  Vl)]  (B.71) 

^  n=l  m=l 

Therefore,  substituting  the  appropriate  derivatives  into  the  update  equations  for  each  parameter 
provides  the  learning  rules  for  a  network  implemented  using  the  MSB  objective  function.  By 
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defining  t)  =  2C/iV,  where  C  is  a  constant,  the  update  equations  for  the  parameters  will  be  as 
follows: 


and 


and 


^MN  =  ^MN  -  niVN  -  dN)yif(l  -  yN)yM 

N 

^LM  =  ^LM  -  V  X^(yn  -  rfn)yn(i  “  yn)t«MnyAf  (1  “  yM)yi. 

n=l 


(B.72) 

(B.73) 


N  M 

^KL  =  ^KL-'l  “  dn)ynil  “  J/n)[X^  “  ym)WLmyL0-  “  (B.74) 

n=l  m=:l 

and 


"  ^N)y7sr(l  -  vn)  (B.75) 

and 

N 

Y^iVn  -  dn)ynil  “  yn)wMnyM{^  -  yw)  (B.76) 

n=l 

and 


N  M 

=  O'!  -  >7  '^{Vn  -  rfn)yn(l  “  yn)[X]  «'mnym(l  “  ym)«»LmyL(l  “  y^)]  (B.77) 

n=l  m=l 

Cross  Entropy  ( CE) 

The  incremental  update  rules  for  the  network  parameters  can  be  found  by  minimizing  the 
CE  function  with  respect  to  the  network  parameters.  The  CE  function  is  defined  as 


Letting  r;  be  a  constant  which  controls  the  learning  rate,  the  update  equations  for  the  parameters 
will  be  as  follows: 


and 


and 


and 


and 


and 


^MN  =  ‘^MN-n 


dCE 

dWMN 


(B.79) 


~  'I 


dCE 

dwiM 


(B.80) 


^KL  = 


dCE 

dtVKL 


(B.81) 


dCE 

dajf 


(B.82) 


dCE 

dcTM 


(B.83) 


crt=crl-  T) 


dCE 

d<Ti 


(B.84) 


First  let’s  find 


acB 


dCE 

dWMN 


1  d 
N  dWMN 


N 

log(y„)  +  (1  -  d„)log(l  -  y„)] 


n=:l 


The  derivative  of  the  log  function  is 


(B.85) 


1 

uln(a) 


DxU 


(B.86) 


Thus 


B-12 


d 


(B.87) 


dCE 

dWMN 


N  dwMN 


log(yiv)  +  (1  -  dn) 


dWMN 


log(l  -  y„)] 


Taking  the  derivative  provides 


dCE  _ _ 1  ,  dN  dyj^r  (1  -  d/y)  3(1  -  y^) 

dwMN  ^n{10)  dwMN  (l-yN)ln(lO)  dtVMN 

This  can  be  written  as 


(B.88) 


dCE  _  1  fd;v  _  (l-djv),  OpN 

dwMN  N  In(lO)  yN  (1  -  Vn)  dwMN 


(B.89) 


This  simplifies  to 


dCE  _  1  rjdN -VNdN -VN +  yNdN),  dy^ 

dwMN  ~  iVln(lO)^  yAr(l-l/Ar)  ^Bwmn 

Which  in  turn  simplifies  to 


(B.90) 


dCE  _  1  r  (d/f  -  yN)  1  dyN 

dwMN  7/  In(lO)  i/iv(l  -  yNydwMN 


Substituting  equation  B.5  for  gives 


(B.91) 


dCE 

dWMN 


1  f(djV-y/v) 
//ln(10)^yAr(l  -2/jv) 


■]yN(i  -  yjv)yAf 


This  in  turn  simplifies  to 


(B.92) 


dCE _ (dAT  -  yN)yM 

dwMN  Wln(lO) 


(B.93) 


Similarly 


dCE  _  1  ..  {dii  —  yN)  1  dVN 

d<TN  ~  N]n{10yyN{l-yN)h(TN 

Substituting  equation  B;28  for  provides 


(B.94) 


dCE 


1  r  (dN  -  Dn) 
Afln(10)^yAr(]  -Dn) 


■][y;v(l  -  yN)] 


(B.95) 
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which  simplifies  to 


dCE  _  {dff  —  ys) 
d(Tff  7V^ln(10) 


(B.96) 


Now  let’s  find 


6CB 

Swi.14  • 


dCE 

dwiM 


1  d 
N  dwiM 


N 

log(yn)  +  (1  -  dn)  log(l  -  Vn)] 


n=l 


(B.97) 


Thus 


Taking  the  derivative  provides 


dCE  ^  dn  dyn  (1-dn)  ^(l-yiv)i 

dwLM  ^  ^  In(lO)  dwiM  (1  -  y»)ln(10)  Owlm 

This  can  be  written  as 


(B.98) 


(B.99) 


dCE  ^  1  .  _  (l-d„),  dvn 

dwLM  //^(lO)  {l-VnYdwtM 

This  simplifies  to 


dCE  _  1  -^Mn-yndn-yn^-yndn),  dyn 

dWLM  iV!n(10)^^  yn(l-yn)  dwiM 

Which  in  turn  simplifies  to 


(B.lOO) 


(B.lOl) 


9GE  _  1  {dn  —  yn)  1  dyn 

dwLM  ~  N  In(lO)  “  yi.(l  -  ytCydwtM 

Substituting  equation  B.15  for  provides 


(B.102) 


dCE 

dwiM 


1 

N  In(lO) 


{dn  -  yn) 
yn(i  -  yn) 


](yn(i  -  yn)WMnyM(l  ~  J/Af)yi] 


(B.103) 


B-14 


This  can  be  written  as 


8CE 

dwiM 


1 

iVln(lO) 


N 

n=l 


(B.104) 


Similarly 


dCE  _ _ 1  (dn  —  Vn)  1  8yn 

daM  ~  iVln(lO)  -  ynydaM 

Substituting  equation  B.38  for  provides 


dCE 

dffM 


1 

7Vln(10) 


(^n  —  Vn) 

yn(i  -  yn) 


■][yn(l  -  yi.)«»Af„yAf  (1  -  yw)] 


(B.105) 


(B.106) 


This  simplifies  to 


dffM 


1 

iVln(lO) 


N 

-  yn)«»MnyAr(i  -  yAf) 

n=l 


(B.107) 


Now  let’s  find 


dCE 

dWKL 


1  a 

iV  5u'kl 


yv 


log(y„)  +  (1  -  dn)  log(l  -  y„)] 

n=r 


(B.108) 


Thus 


^  '-sb.) + (1  -  ‘'.>5;^  >oe0  -  (B-io^) 

Taking  the  derivative  provides 

_ _ 1  Y^r  dyn  (1  —  dn)  ^(1  —  yn)^  /'Rlin^ 

dwKL  fV  J^^y„  In(lO)  (l-yn)ln(lO)  Bwkl  ^  \  > 

This  can  be  written  as 


dCE  _  1  _  (1  ~  ^")if  ] 

dWKL~  ^ln(10);^yn  (1-yn) 


(B.lll) 


B-15 


This  simplifies  to 


N 

dCE  _  1  '^Mn-yndn-yn-\-yndn)y  dy„ 

dWKL  2/(i(l-J/n)  dWKL 

Which  in  turn  simplifies  to 


dCE  _  1  Ar(rfn-yn).  dyn 

dWKL  Ann(lO)  yn(l  -  Vn)  dWKL 


(B.112) 


(B.113) 


Substituting  B.23  for  gives 


dCE 

dWKL 


1 

i\fln(10) 


N 


E 


{dn  -  Vn) 
yn(l  -  Vn) 


M 

yn(l-yn)[S 

m=l 


^mn  ym(i  -  ym)wLmyL0-  -  yi)y/f] 


(B.114) 


This  in  turn  simplifies  to 


dCE  1 

^  "ivInTioi  W'mnymCl  -  ym)wLmyL{l  -  yi)yK] 

'  n=l  vh=l 

Similarly 

dCE  _  1  {dn  —yn)  dyn 

dcTL  ~  AT  In(io)  y»(l  -  yn)  dai 

Substituting  equation  B.46  for  provides 


(B.115) 


(B.116) 


dCE 

d<Ti 


1 

Wln(lO) 


N 


E 


{dn  ~  yn') 
yn(l-yn) 


M 

yn(i  -  yn)[^  iymnym(i  “  ym)u'Lmy£,(i  “  y^)] 

m=l 


(B.117) 


which  simplifies  to 


dCE 

dai 


I  N  M 

]Vbp)  “  yn)[Y^  Wmnym(l  -  ym)u;LmyL(l  “  yi)] 


(B.118) 
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Letting  t]  —  C/[Nln{lO)]  where  C  is  a  constant,  the  u  iate  equations  for  the  network  parameters 
will  be' 


^tfN  =  ^MN  +  -  yiv)yAf] 


(B.119) 


and 


N 

+  n  “  yn)«'Mnyilf  (1  -  yM)yL] 

n=l 


(B.120) 


and 


N  M 

^KL  =  ^KL  +  »?  ~  ^mnymO-  “  “  yL)yK] 

n=l  m=l 


(B.121) 


and 


~  Vn) 


(B.122) 


and 


N 

=  +  ^  -  yn)t«A/nyAf (1  “  yw) 

n=l 


(B.123) 


and 


N  M 

=  Y^^^  ~  «'«»n(ym(l  -  ym)wLmyLil  “  yi)]  (B.124) 

n=l  m=l 


B.5  Classificaiion  Figure  of  Merit  (CFM) 

The  update  rules  for  the  network  parameters  can  be  found  by  maximizing  the  CFM  function 
with  respect  to  the  network  parameters.  The  classification  CFM  objective  function  is  defined  as 

1  ^ 


where 
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yn  =  output  of  an  incorrect  classification  node 
l/c  =  output  of  the  correct  classification  node 
a  =  a  sigmoid  scaling  parameter 
=  a  sigmoid  discontinuity  parameter 
^  =  a  sigmoid  lateral  shift  parameter 
Furthermore,  let 


Here  Zn  is  a  function  of  both  the  correct  and  incorrect  node  outputs.  Therefore,  the  CFM  objective 
function  can  be  written  as 


CFM^-^  E  «^n(yn,yc)  (B.127) 

Letting  be  a  constant  which  controls  the  learning  rate,  the  incremental  update  equations  for  the 
parameters  can  be  found  from  the  following  equations: 

.  ^  dCFM 

and 


^LM  —  ^LM  + 


dCFM 

dwLM 


and 


w 


+ 

KL 


^KL  +  ^ 


dCFM 

dwKL 


and 


(B.129) 


(B.130) 


and 


'N 


=  <^N  +  n 


dCFM 

dan 


(B.131) 
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(B.132) 


=  +  r? 


dCFM 

d(TM 


and 


=  cr£  +r? 


dCFM 

dffL 


Now,  taking  the  following  derivatives 


dzn  _  _5_ 
dyn  ~  dyn 


[1 


^  g(-/5yc+/?yn+()j-i 


Simplifying 


^  g(-^yc+/3yn+C)]-2_^rg(-/Jyc+/?yn+()i 

^J/n  dyn 


(B.133) 


(B.134) 


(B.135) 


This  can  be  written  as 


(B.136) 


which  simplifies  to 


F)r 

^  =  (B.137) 

Similarly 

^  =  Ah  +  g(-/?yc+/Jy»+C)]-i  (B.138) 

^Vc  dye 

Simplifying 


^  =  _[1  +  g(-/3y=+^yn+C)]-2^rg(-^yc+/Jyn+C)i  (B.139) 

aye  dye 


This  can  be  written  as 


_  ^g(-/3yc+/3yn+()[i  +  g(-i3yc+/Jyn+C)]-2 


(B.140) 


Simplifying  again 


B-19 


(B.141) 


First,  let’s  find  the  update  rules  for  a  weight  linking  node  M  in  layer  2  to  an  incorrect  classification 
node  Ndn  layer  3,  wmn. 


dCFM  _  ft  A  dZn 
dWMff  N  -1  dwMN 

But  ZN  =  f{ye,yn)  therefore, 

dCFM  _  a  dzfi  dytf 
dwMN  N  —  I  ayif  dwMN 

Substituting  equation  B.5  for  and  equation  B.137  for  provides 

~  “  z/v)y/v(i  -  yiv)yAf  (B.144) 


(B.142) 


(B.143) 


Similarly 


dCFM  _  or  dzN  dyj^ 
d(Tj^  N  —  \  dyi^  d<Ti^ 


(B.145) 


Substituting  equations  B.28  for  and  equation  B.137  for  provides,  for  an  incorrect  classi¬ 
fication  node, 


~  «jv)yjv(i  -  yw)  (B.146) 

For  the  correct  classification  node,  the  needs  to  be  found  to  maximize  the  CFM  objective 

function,  where 


dCFM 

dwMC 


N~1  ^  dwMC 


This  can  be  simplified  to 


dCFM  _  a  dzn  dye 

dwMc  ~  N  dye  dwMC 


(B.147) 


(B.148) 
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Substituting  equation  B.5  for  and  equation  B.141  for  ^  provides 


dCFM 

dtOMC 


aP 

IT^ 


E 


n=lnifcc 


zn{\  -  i^n)yc(i  -  yc)yM 


Similarly 


dCFM  _  Of  dzn  dye 

d<Tc  iV^  —  1  dye  d<Te 

W  — J  n^c 


Substituting  equation  B.28  for  and  equation  B.141  for  provides 


dye 


N 


dCFM  a0  f  \  /i  \ 

L  ^«(i-^n)yc(i-yc) 


Now  let’s  find  the 


This  can  be  written  as 


dCFM 

dwLM 


a 

JT^ 


N 


E 


dZn 

dWLM 


dCFM  _  /  Of  ^  .dzn  dyn  dzn  dye  . 
diVcM  N  -I  ^  dyn  dwiM  dye  dwiM 


(B.149) 


(B.150) 


(B.151) 


(B.152) 


(B.153) 


Substituting  equation  B.15  for  and  5^^  and  equations  B.137  and  B.141  for  and 
gives 


dCFM 

dwjjM 


Or/? 

N-1 


N 

£  ^"(1 


W  =  ln9tc 


Zn)[yc(l  -  yc)wMc  -  yn(l  -  yn)t«Afn]yM(l  -  yM)yL 


(B.154) 


Similarly 


dCFM  _  a  ^  dz„  dyn  dzn  dye  . 

dcTM  N-1  ^  ^dyn  doM  dye  daM 

Substituting  B.38  for  and  and  equations  B.137  and  B.141  for  and  gives 
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^n(l  -  ^n)[j/c(l  -  ye)»)Me  “  yn(l  “  yn)tt>A/n]yA/(l  -  yAf)  (B.156) 


dCFM 

dffM 


a(3 


N 


E 

Initc 


Now  let^s  find  the 


dCFM 
Owkl  * 


dCFM  g  ^  dzn 
dWKL  ~  N-1  dWKL 


This  can  be  written  as 


(B.157) 


dCFM  _  a  ^  .dzn  dyn  dz„  dye  . 

dWKL  N -I  dyn  dwKL  ^  %  dWKL  ’ 


(B.158) 


Substituting  equation  B.23  for  and  and  equations  B.137  and  B.141  for  and 
gives 


dCFM 

dWfCL 


afi 

N-l 


N 


2n(l 


M 

^n)[yc(l  -  yc)  Wmc 

m=l 


Af 

-  yn(l  -  yn)  X)  «;m„]ym(l  -  ym)«'£myi(l  -  yL)yK 

m=l 

Similarly 

dCFM  _  a  ^  dzn  dyn  dzn  dye 
dai  AT  - 1  “  dyn  dat,  dye  dcxi 

Substituting  B.46  for  and  and  equations  B.137  and  B.141  for  and  gives 


(B.159) 


(B.160) 


dCFM 

dai 


or/? 


N 


M 


N 


zri  IZ  ^n(i-^n)[yc(i-yc)X)"''"c 


m=l 


M 


-  yn(l  -  yn)  «>mn]ym(l  “  ym)wLmyLil  “  yi) 


(B.161) 


>n=l 

Thus,  if  7;  =  Ca/S/{N  —  1)  where  C  is  a  constant,  the  update  equation  for  a  weight  linking  node  M 
to  an  incorrect  classification  node  N,  wmn  is 
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^tfN  =  ~  “  ^Ar)yiv(i  -  yN)yM  (B.162) 

and  the  update  equation  for  the  offset  of  an  incorrect  classification  node  N,  cr/sr,  is 

~  ^/sr)yAr(l  ~  yisr)  (B.163) 

The  update  rule  for  the  weight  linking  node  M  in  layer  2  to  the  correct  node  G  in  layer  3  is 

N 

^tlC  =  ^MC+^  X)  2nO--Zn)ycil-yc)yM  (B.164) 

n=ln,tc 

The  update  rule  for  the  offset  of  the  correct  node  C  in  layer  3  is 

0’c  =  <^c  +  ’?  X)  ^"(1  “  ^n)j/c(l  -  yc)  (B.165) 

n  =  ln/c 

The  update  equation  for  a  weight  linking  node  L  in  layer  1  to  node  M  in  layer  2,  wim  as 


N 

+  V  X)  -  Zn)[ycii  -  yc)wMe  -  yn{l  -  yn)wMn]yMii^  -  yM)yL  (B.166) 
while  the  update  equation  for  the  offset  of  node  M  in  layer  2,  (tm  is 


AT 

=  +  V  X^  ZnO-- Zn)[ycil-yc)wMc-ynO--yn)WMn]yM{i-yM)  (B.167) 

The  update  equation  for  a  weight  linking  node  K  in  layer  0  to  node  L  in  layer  1,wk£,  is 


N  M 

^KL  =  ^KL  +  V  “  -2'n)(ye(l  “  ^c)  X^  Wmc 

n=l„^c  m=l 

M 

-  yn(l  -  yn)  X3  t«mn]ym(l  “  ym)«»LmyL(l  “  yL)yK  (B.168) 

m=l 
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M+ 


while  the  update  equation  for  the  offset  of  node  L  in  layer  1,  ai  as 


N 

=  <^l  +  v  -  ^n)[yc(l 


M 

Vc)  Wmc 
m=l 


M 

-  yn(l  -  Vn)  X]  «»mn)ym(l  “  ym)t«£my£(l  “  yt) 
m=l 


(B.169) 
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Appendix  C.  Parzen  Window/Radial  Basis  Function  Relationship 

C.l  Iniroduciion 

This  appendix  will  discuss  the  relationship  between  the  Radial  Basis  Function  and  Parzen 
Window  approach  to  the  estimation  of  a  probability  density  function  from  analysis  of  a  set  of  data 
points. 


CJJ  Density  Estimation  The  Parzen  Window  estimate  of  a  conditional  probability  density 
function,  P{x/Gj)y  provides  a  smooth  estimate  of  the  density  function  from  a  set  of  sample  data 
points  by  assuming  that  each  value  of  the  data  occurring  in  the  sample  set  also  raises  the  probability 
of  any  value  occurring  close  to  that  value  of  the  data.  By  centering  a  kernel  function  at  each  of 
the  data  points,  the  final  value  of  the  estimate  can  be  obtained  by  summing  together  all  the 
contributions  from  each  value  of  the  sample  data.(8:162)  That  is  the  Parzen  Window  estimate  has 
the  form 


.  Nj  . 

where 

K  =  the  number  of  dimensions 

Nj  =  the  number  of  data  points  in  class  J 

h{Nj)  =  a  function  of  Nj  commonly  referred  to  as  the  window  width. 

^(x)  =  the  kernel  estimate  function. 

Thus,  to  form  a  Parzen  Window  estimate  of  the  density  function  from  a  set  of  known  points,  a 
kernel  function  and  a  window  width  have  to  be  chosen.  While  the  window  width  can  be  somewhat 
arbitrary,  the  kernel  function  must  fulfill  several  conditions. 

CJ.2  Conditions  For  the  Parzen  window  estimate  to  work,  the  following  conditions  must 
hold  true  (7:174): 


yoo 

/  =  1 

J-OO 

(e.2) 

.00 

/  \<f>{u)\du  <  00 

/-oo 

(C.3) 

C-1 


Sup\<f>{u)\  <  oo 

(C.4) 

lim  |ti^(tx)l  =  0 

tl^OO  *  ' 

(C.5) 

lim  h^(N)  =  0 

(C.6) 

lim  Nh^(N)  =  oo 

(C.7) 

lim  =  oo 

iV^co  ' 

(C.8) 

If  all  these  conditions  are  met,  the  Parzen  estimate  is  asymptotically  unbiased  and  consistent  at  all 
the  continuous  points  of  P(x/G/)  (7:173-175). 

C.2  Kernel  Selection 

One  type  of  kernel  function  which  satisfies  these  conditions  is  the  gaussian  radial  basis  function 
^{\\x  —  Xnll)  =  [2Tra^h{N)]^^e  1  (C.9) 


where 

a  =  Constant  defining  the  window  width 
h{N)  =  =  a  function  of  N 

N  =  Number  of  data  points 
K  =  Nojmber  of  dimensions. 

C.3  Proofs 

The  proofs  that  the  gaussian  radial  basis  function  meets  the  requirements  for  the  Parzen 
Window  are  shown  below.  In  these  proofs,  the  following  substitutions  have  been  made  to  simplify 
the  algebra.  Let 


C-2 


2(T^h{N) 


(C.IO) 


«*  =  i^k  ~  Xknf  =  (Xjfcn  -  Xkf 


(C.ll) 


and  thus 


«^(p-snii)=(-rvcLo(“0^] 


(C.12) 


The  first  condition  the  kernel  function  must  meet  is  that 


/OO  fOQ 

...  (l>{u)dui...duK  =  l 

•00  J-kO 


substituting  the  equation  for  the  kernel  provides 


which  can  be  written  as 


(C.13) 


f  ...  f  <j){v.)dui...duK  =  (—)~^  f  ...  /  (C.14) 

J  —  OO  — 00  ^  J— 00*/  —  00 


/OO  1*00  pOO  /•OO 

. . .  /  ^(u)dui . .  .du/r  =  (-)“^[2  /  dui] . .  .[2  /  (C.15) 

•  00  •/— 00  ®  */o  Jo 


now  from  (25:640) 


e-^'^^du 


1  Fir 
~  2y  a 


(C.16) 


substituting  into  equation  C.15  provides 


(C.17) 


Therefore  the  equation  C.2  is  satisfied.  The  second  equation  that  must  be  met  is  that 


substituting  the  equation  for  the  kernel  function  provides 


J  *  J  . .  .duK  <  oo  (C.19) 

Since  equation  C.2  is  satisfied,  then  so  is  equation  C.3.  Equation  C.4  describes  the  third  condition 
that  must  be  met.  For  this  condition,  the  supremum,  or  least  upper  bound  must  exist.  That  is 

5«p|^(u)|  <  00  (C.20) 


Substituting  for  |^(u)|  reveals 


Sup|^(«)|  =  S«p|(-)-^e-Er  (C.21) 

a 

The  maximum  value  for  |<^(ti)l  is  unity.  Therefore 

Sup\(f>{u)\  =  1  <  oo  (C.22) 

Thus,  equation  C.4  is  satisfied.  Equation  C.5  describes  the  fourth  condition  that  must  be  met.  For 
this  condition 


lim  \u(j>(u)\  =  0  (C.23) 

u~*oo 

Substituting  the  equation  for  the  kernel,  and  suppressing  the  constant,  (7r/a)“*^/^,  provides,  for 
the  dimension 


now 


lim  \uk^(uk)\  =  lim 

«-->oo  u-*oo 


(C.24) 


lim  =  lim  I 

tifc-+00  «*-»oo 


e<i(u*)’ ' 


(C.25) 


using  L’Hopital’s  rule 


(C.26) 


lim 


Uk 


lim 

xtk-*oo 


1 

2auke<^^)'' 


=  0 


Thus,  this  condition  is  satisfied.  Equation  C,6  describes  the  fifth  condition  that  must  be  met.  Here, 


\\rt^h^{N)  =  0  (C.27) 

must  be  satisfied.  Substituting  the  equation  for  h(^)  provides 

lim  h^(N)  =  lim  (C.28) 

N-*co  '  ’  AT-oo^  ■'  '  ' 

Simplifying  the  equation  shows 

lim  -  lim  =  0  (C.29) 

iV-oo  '■  ^  iV-oo  '  ’ 


Thus,  this  condition  is  satisfied.  Equation  C.7  describes  the  sixth  condition  that  must  be  met.  For 
this  condition, 


lim  =  oo 

;v-oo  ^  * 


(C.30) 


must  be  met.  Substituting  the  equation  for  h(N)  provides 


lim  Nh^(N)=  lim  N(N^f 

N-*oo  ^  ^  AT-oo  ^ 


(C.31) 


This  can  be  written  as 


lim  Nk^(N)  =  lim  AT-®®  =  oo  (C.32) 

Thus,  this  condition  is  met.  Equation  C.8  describes  the  seventh  condition  that  must  be  met.  For 
this  condition, 

Jim  Nh{N)^  =  oo  (C.33) 

Substituting  the  equation  for  h{N)  shows 
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Jir^  Nh{Nf  = 


This  can  be  written  as 


lim  Nh(N)^  =  lim 

N-*oo  '  ^  AT-oc  ' 

which  can  be  simplified  to 


lim  Nh(N?  =  lim  =  oo 

AT-^co  ^  '  N-^oo 


Thus,  this  condition  is  fulfilled. 

By  meeting  these  conditions,  the  guassian  radial  basis  function  can  be  used 
conditional  probability  distribution  from  a  set  of  sampled  points.  In  this  estimate, 


moj) = ^5^ 

now,  since  h{N)  =  m  1  even  for  large  values  of  N,  the  equation  becomes 


1  ^  K  <£tZ£ifl£l 


U 


(C.34) 


(C.35) 


(C.36) 

to  estimate  a 


(C.37) 


(C.38) 
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Appendix  D.  Kernel  Classifier  Network  Training  Algorithms 

D.l  Iniroduciion 

In  this  section,  the  training  algorithms  for  a  radial  basis  function  network  will  be  derived. 

Consider  the  feedforward  artificial  neural  network  as  shown  in  figure  3.7  with  the  following 
parameters:. 

Layer  0  -  input  layer  with  K  possible  nodes 
Layer  1  -  hidden  layer  with  L  possible  nodes 
Layer  2  -  output  layer  with  M  possible  nodes 

Let  the  weights  between  layers  be  defined  as  follows: 
wjci  -weight  linking  node  k  in  layer  0  to  node  /  in  layer  1. 

^im  -weight  linking  node  I  in  layer  1  to  node  m  in  layer  2. 

Let  the  transfer  function  for  the  nodes  in  each  layer  be  as  follows: 

Layer  0  -  yjb  =  xjb  =  identity  function 

24^2  J  =  gaussian  radial  basis  function 
Layer  2  -  =  12hzi  ^imVi  ==  linear  function 

Thus  the  parameters  of  weights  wl  and  wm,  and  the  spreads  (Tl  will  characterize  the  network. 

D.2  Incremental  MSE  Minimization 

One  method  of  determining  these  network  parameters  is  to  use  the  method  of  incremental 
backpropagation  according  to  the  MSE  objective  function  defined  here  as 

M 

MSE=:  —  Y,{ym-dmf  (D.l) 

m=l 

A  network  developed  using  the  MSE  objective  function  seeks  to  have  a  minimum  error  over 
all  patterns  with  respect  to  the  network  parameters.  The  incremental  update  equations  for  the 
network  parameters  are  defined  as 


wIm  =  ^LM  -  n 


dMSE 

dwiM 


(D.2) 


and 


D-1 


+  ■  _  dMSE 


+  _  dMSE 

^KL  -  <^KL  -  r)-^ 


Here  77  IS  3>  constant  which  controls  the  rsite  of  updsitc*  Let’s  first  minimize  the  error  with 
respect  to  a  specific  weight,  say  linking  node  L  in  the  hidden  layer  to  node  M  in  the  output 
layer. 


dMSE  _  5  1  A  2 

dwLM  ~  dWLM^M  ~  J 


therefore 


dMSE  l^d  2 

dWLM  “  A/  dWLM^'^”' 

TTX—l 


dMSE  _  2  _  dyM 
dWLM  M  dWLM 


Vm  =  'y',wmyt 

i=i 


dwLM 


therefore 


dMSE  2, 

-^  =  M^yM-dM)yL 


(D.IO) 


If  jj  =  lCjM  where  C  is  a  constant,  the  training  rule  for  this  weight  is 
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-  niVM  -  dM)yL 


(D.ll) 


Now,  let’s  minimize  the  error  with  respect  to  the  weight,  or  center,  of  radial  basis  function  L  in 
layer  1  providing  the  link  to  node  K  in  layer  0,  wkl- 


dMSE 

dWKL 


a  1 

m=l 


(D.12) 


or 


dMSE 

dWKL 


_1_ 

M 


E 


msl 


d 

dWKL 


(D.13) 


thus 


but 


dMSE  2  ^  ,  dym 

dwKL  ~  M 

m=l 


(D.14) 


L 

ym-Y^Wimyi 

1=1 


therefore 


or 


but 


or 


^Vm  _  d 
dwKL  dwKL  " 


dym  _  dyi 
dWKL  ^^'^dWKL 


K 

1^=1 


(D.15) 


(D.16) 


(D.17) 


(D.18) 
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therefore 


dvL 

dxOKL 


Simplifying 

^VL  _  ^-r  i?^K-WKLT, 

dwKL  dw^t^  2<r^^ 

or 


(D.20) 


(D.21) 


dyi,  _  {xK  -  wkl) 
dwKT.  ^ 

Combining  equations  D.i4,  D.17,  and  D.22igives 


(D.22) 


dMSE 


M 


m=l 


{XK  -  ^Kl) 


^KL 


(D.23) 


If  r}  =  2C/M  where  C  is  a  constant,  the  update  equation  for  the  centers  of  the  radial  basis  functions, 
or  weights  linking  the  input  layer,  layer  0,  nodes  to  the  hidden  layer,  layer  1,  nodes,  can  be  written 


as 


m=l  ^KL 

Now  let’s  minimize  the  error  with  respect  to  the  spread  of  the  radial  basis  function  in  the 
direction  of  the  node  in  the  input  layer,  layer  0,  cr/fi. 


dMSE 

dffKL 


M 


(D.25) 


or 


dMSE 

dcTKL 


1  " 


(Vm  ~  dm)^ 


(D.26) 


or 


but 


dMSE 

dffKL 


M 


-  "m 


m=l 


dym 

dcTKL 


L 

Vm  =  ^WimVl 
J=1 


therefore 


dym  A  % 

3o-k  t  Scryfi 


This  expands  to 


but 


or 


therefore 


dym  _  dyi 
d<7KL  dcTKl 


yi  = 


yL  =  e 


(D.27) 


(D.28) 


(D.29) 


(D.30) 


(D.31) 


(D.32) 


(D.33) 


Simplifying 


_  e"ELi  ^  r  (=gK- - 

5<TK£.  ScT/ft 


(D.34) 


or 
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d(TKL 

Combining  D.27,  D.30,  and  D.35  gives 
dMSE  2 


dyi  _ {xK  -  rvKi,y 


'KL 


(D,35) 


_  2  j  ^ . {xk  —  wklY 

—  T7  /  y(y>n  ~  dni)wLmyL  3 


doTKL  M  VKi 

Letting  r\  —  2CfM  where  C  is  a  constant,  the  incremental  training  rule  can  now  be  defined  as 


(D.36) 


+  -  /  j  ^  (®/f  “  ^kl)^ 

^KL  =  (^KL-^2.  \y>n  -  dm)WLtnyL - - - 

m=l  ^KL 


(D.37) 


D.S  Incremenial  Average  Update 

In  this  section,  the  update  rule  for  keeping  the  centers,  or  weights,  of  the  radial  basis  functions 
at  the  average  of  the  patterns  within  their  assigned  class  will  be  derived.  Let  be  the  previous 
average  of  the  N  pattern  vectors  in  the  cluster  at  time  i.  Then 

1  ^ 

=  (D.38) 

n=l 

Similarly 


This  can  be  written  as 


1)  = 


1 

N  +  l 


N+l 


n=l 


(D.39) 


wt(t  + 1)  = 


1 

N  +  l 


N 

(H)  + ^w+0 

n=l 


Which  in  turn  can  be  written  as 


(D.40) 


This  reduces  to 


N+r  N 


+  ®/V+l) 


(D.41) 
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Wi{t  + 1)  =  +  *Ar+il 


(D.42) 


Further  simplifying 


TO((  +  l)  =  (l--j^)®,(0  +  f^  (D.43) 

This  reduces  to 


Wl(t  +  1)  Wl{t)  +  (D.44) 

D.4  Global  MSE  Minimization 

In  this  section,  the  technique  of  globally  minimizing  the  MSE  objective  function  throug*  the 
use  of  a  matrix  inversion  will  be  established  (23).  Define  the  total  error  due  to  all  input  training 
patterns,  P,  as 


-  p  M 

MSE^-Y,  (D-45) 

p=:l  m=l 

where  dpm  is  the  desired  value  for  the  output  node  due  to  the  pattern  and  ypm  is  the  actual 
value  for  the  output  node  due  to  the  p^^  pattern.  This  error  must  be  minimized  with  respect 
to  the  weights  connecting  a  particular  node,  say  node  B,  in  the  hidden  layer,  to  a  particular  node, 
say  node  D,  in  the  output  layer.  That  is,  the  error  can  be  minimized  by  setting 


but 


or 


dMSE 

dwBD 


(D.46) 


dMSE 

dwsD 


d 


P  M 


dwpD  ^2 


4  4  AVA 


p=l  m=l 


(D.47) 


dMSE 

dWBD 


p 

=  Y^yp^  ~  ^p^^ 

p=i 


dypD 

dWBD 


(D.48) 
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but 


Here,  yp\  is  the  output  of  the  radial  basis  function  node,  where  1  <  /  <  L,  due  to  the  pattern 
and  PpB  is  the  output  of  the  radial  basis  function  due  to  the  pattern.  This  allows  the 
following  equation  to  be  written: 


L  p 

widMib  =  Y  ^p^VpB  (D.57) 

P=i 

Now,  the  MSE  can  be  minimized  by  ensuring  the  dMSE/dwim  —  0  for  all  weights.  Define  a  weight 
matrix  W  as  an  L  by  M  matrix  as 


Wn  Wi2  ...  WxM 
^21  ^22  •  •  ■  ^2M 

m2  •  • .  mM 


(D.58) 


Minimizing  the  MSE  is  the  same  as  making_5M5jE/5iy  =  0  for  all  weights.  That  is,  the  error  is 
minimized  by  setting  the  dMSE/dwim  =  0  for  each  weight  in  the  weight  matrix.  But,  from  above, 
for  a  given  weight  wbD}  the  dMSE/dwsD  =  0  when 


L  p 

Y  ^idMib  =  Y  ^pBVpB  (D.59) 

;=i  p=i 

This  equation  states  that  the  derivative  of  the  error  with  respect  to  a  weight  linking  the  node 
radial  basis  function  in  the  hidden  layer  to  the  node  in  the  output  layer  is  minimized  when  the 
sum  of  all  the  weights  in  the  hidden  layer,  multiplied  by  the  summation  of  the  product  of  the  radial 
basis  function  outputs  and  the  B^^  radial  basis  function  output,  summed  over  all  patterns,  is  equal 
summation  of  the  product  of  the  desired  output  of  the  output  node  and  the  B^^  radial  basis 
function  summed  over  all  patterns.  For  example,  when  5  =  l,i)  =  1,  dMSE/dww  =  0  implies 
that 


L  p 


Y^wnMn  =  Yldpiypi 

1=1  p=i 


(D.60) 


which  expands  to 


wnMn  +  W21M21  +  . . .  +  =  dii2/n  +  + .  ■  ■  +  dpij/pi  (D.61) 
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Also,  when  B  =  1,  B  =  2,  then  dMSE/dwx2  =  0  implies  that 


^wi2Mn  =  Y^dp2ypi  (D.62) 

/=i  p=i 

which  expands  to 


tvi2Mii  + 16*22^21  +  . . .  +  =  d^yii  +  ^222/21  +  .  • .  +  dp2ypi  (D.63) 

Finally,  when  B  =  1, 1)  =  2,  then  dMSE/dw2i  =  0  implies  that 

L  p 

Y^wnMt2  =  Y^dpiyp2  (D.64) 

1=1  p=i 

which  expands  to 


y^nMi2  +  ti;2i^22  +  . . .  +  wiiMl2  =  dnyi2  +  ^212/22  +  •  •  •  +  dpi2/P2  (D.65) 

and  so  forth  for  each  of  the  weights  connection  nodes  in  the  hidden  layer  to  nodes  in  the  output 
layer.  This  gives  a  set  of  L*M  equations  with  L*M  unknown  weights  which  can  be  written  as 

M^W  =  Y^S  (D.66) 

Here  M  is  an  L  by  L  matrix  containing  the  summation,  over  all  patterns,  of  the  product  of  each 
radial  basis  function  output,  for  a  given  input  pattern  and  the  B*^  radial  basis  function  output  for 
that  pattern.  That  is 


Mn 

Mi2 

...  Mii 

M21 

M22 

. . .  Mai 

Mil 

Ml2 

. . .  Mii 

(D.67) 


where  Mw  =  Ef=i  VpWpB 

Also,  W  is  an  L  by  M  matrix  containing  the  weights  linking  the  nodes  in  the  hidden  layer  to  an 
nodes  in  the  output  layer.  That  is 
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wn 

^12 

•  •  •  WiM 

W=: 

W2l 

^22 

•  -  •  W2M 

(D.68) 

^  ^L1 

...  WiM 

where  wbd  is  the  weight  linking  the  node  in  the  hidden  layer  to  the  node  in  the  output 

layer.  Here,  Y  is  a  P  by  L  matrix  containing  the  outputs  for  each  of  the  L  radial  basis  functions 

for  all  P  patterns.  That  is 

yn 

yi2 

•  •  •  2/l£ 

y  = 

2/21 

2/22 

■  •  •  y2L 

(D.69) 

.  ypi 

yp2 

■  ■  •  ypL 

Finally  S  is  a  P  by  M  matrix  containing  the  desired 

outputs  for  each  of  the  M  output  nodes  for  all 

P  patterns.  That  is 

dx\ 

di2 

...  d\M  1 

S  = 

C?21 

^22 

... 

s; 

(D.70) 

dpi 

dp2 

•  •  •  dpM 

Thus,  the  weights  which  minimize  the  MSE  can  be  found  by  using  the  following  equation: 


Now,  the  optimized  weight  Wsp  can  be  found  as: 


or 


=  N  = 


nil 

n\2  . 

..  nij^ 

«21 

7122  • 

•  •  7l2L 

riLX 

nL2  . 

..  n^L 

(D.71) 


(D.72) 
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=  N{Y'^S)bd 


(D.73) 


Now 


I2p=zi  2/pi^pi  ypi^p2 

J2p=i  l/p^dpi  ]Cp=i  Vp2dp2 


Sp=l  Vpl^pM 

52^=1  Vp^dpM 


2f=l  VpLdpl  X2p-1  ypLdp2 


Hpz^iypidpM  J 


therefore: 


(D.74) 


^BD 


L  P 

Yy^ypidfD)NBi 


t=l  P=1 


(D.75) 


This  method  only  works  for  matrices  that  do  not  become  singular  or  near-singular,  which  can 
happen  if  the  exemplar  data  points  used  to  center  the  radial  basis  functions  contain  redundant 
information.  If  they  do,  the  Singular  Valued  Decomposition  of  the  matrix  may  be  used.  Conversely, 
the  <X8  of  the  offending  nodes  may  be  adjusted  to  eliminate  the  redundancy. 
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Appendix  E.  Tables  for  Data  Analysis 


EJ  Iniroduciion 

This  appendix  contains  the  data  obtained  from  the  neural  network  testing,  discussed  in  Chap¬ 
ter  4,  for  the  communications  data  and  the  radar  signal  data.  This  appendix  begins  with  the  com¬ 
pilation  of  the  data  for  the  Kernel  Classifier  and  Hyperplane  Classifier  networks  implemented  to 
categorize  the  communications  data.  This  is  followed  by  a  compilation  of  the  data  for  the  Kernel 
Classifiers  used  to  categorize  the  radar  data. 


E,2  Communicaiions  Signal  Characterization 

This  data  consisted  of  202,  50-dimensional  pattern  vectors.  These  pattern  vectors  represented 
either  a  direct  sequence  or  a  linear-stepped  frequency-hopped  digital  communications  signal.  Both 
Hyperplane  and  Kernel  Classifier  networks  were  developed  to  categorize  this  data. 

E,2,l  Hyperplane  Classifiers  The  topology  for  the  sigmoidal-based  Hyperplane  Classifier 
networks  implemented  for  this  problem  is  shown  in  figure  3.3.  The  network  parameters,  being  the 
weights  and  the  offsets,  were  set  using  the  backpropagation  algorithms  developed  in  Chapter  3. 

E.2.L1  MSE  Algorithm  Table  E.l  shows  the  categorization  performance  for  ten  Hy¬ 
perplane  Classifier  networks  whose  parameters  were  trained  using  the  incremental  backpropagation 
algorithm  according  to  the  MSE  objective  function.  The  data  seed  matched  the  run  number  and 
51  training  vectors  were  loaded  for  each  class.  The  weight  seed  and  sigma  seed  were  both  zero  and 
the  record  seed  was  one.  The  network  had  50  nodes  in  layer  0,  18  nodes  in  layer  1,  ten  nodes  in 
layer  2  and  two  nodes  in  layer  3.  Each  of  the  nodes  was  assigned  the  sigmoidal  transfer  function. 

E,2.L2  CE  Algorithm  Table  E.2  shows  the  categorization  performance  for  ten  net¬ 
works  whose  parameters  were  trained  using  the  incremental  backpropagation  algorithm  according 
to  the  CE  objective  function.  The  data  seed  matched  the  run  number  and  51  training  vectors  were 
loaded  for  each  class.  The  weight  seed  and  sigma  seed  were  both  zero  and  the  record  seed  was  one. 
The  network  had  50  nodes  in  layer  0,  18  nodes  in  layer  1,  ten  nodes  in  layer  2  and  two  nodes  in 
the  layer  3.  Each  node  was  assigned  the  sigmoidal  transfer  function. 

E.2,LS  CFM  Algorithm  Table  E.3  shows  the  categorization  performance  for  ten  Hy¬ 
perplane  Classifier  networks  whose  parameters  were  trained  using  the  incremental  backpropagation 
algorithm  according  to  the  CFM  objective  function.  The  data  seed  matched  the  run  number  and 
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Table  E.2.  CE  Network  Performance 
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Table  E.3.  CFM  Network  Performance 


51  training  vectors  were  loaded  for  each  class.  The  weight  seed  and  sigma  seed  were  both  zero  and 
the  record  seed  was  one.  The  network  had  50  nodes  in  layer  0,  18  nodes  in  layer  1,  ten  nodes  in 
layer  2  and  two  nodes  in  layer  3.  Each  node  was  assigned  the  sigmoidal  transfer  function. 


E.2.2  Kernel  Classifiers  The  topology  for  the  radial  basis  function  Kernel  Classifier  net¬ 
works  implemented  for  this  problem  is  shown  in  figure  3.7  The  network  parameters,  being  the 
weights  and  the  spreads,  were  set  using  the  algorithms  developed  in  Chapter  3. 


E.2.2J  Nodes  at  Data  Points  Tables  E.4  and  E.5  show  the  training  and  test  catego¬ 
rization  performance  for  a  Kernel  Classifier  network  with  a  variable  number  of  hidden  layer,  layer 
1,  nodes.  The  weights  for  these  nodes  were  established  using  the  Nodes  at  Data  Points  algorithm. 
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Table  E.4;  Nodes  at  Data  Points  Training  Performance  v 
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Table  E.5.  Nodes  at  Data  Points  Test  Performance  vs  Nodes 


For  the  software  parameters,  the  output  threshold  was  set  to  one,  the  sigma  threshold  was 
set  to  four.  The  training  rule  for  the  sigma  was  the  Scale  Sigma  According  to  Class  Interference 
with  interference  threshold  of  .4.  The  data  seed  was  zero  and  51  training  vectors  were  loaded  for 
each  class.  The  number  of  nodes  in  layer  0  was  50,  while  the  number  of  nodes  in  layer  1  is  shown 
in  the  table  and  the  number  of  nodes  in  layer  2  was  two.  The  transfer  function  for  the  nodes  in 
layer  1  was  gaussian  while  the  transfer  function  for  the  nodes  in  layer  2  was  linear.  The  weights 
linking  layer  1  nodes  to  layer  2  were  set  via  global  minimization  of  the  MSE. 
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E.2,2,2  Kohonen  Training  Tables  E.6  and  E.7  show  the  training  and  test  categoriza¬ 
tion  performance  of  Kernel  Classifier  networks  with  the  layer  1  weights  trained  using  the  Kohonen 
Training  algorithm  with  the  RBF  spreads  set  using  the  P-Nearest  Neighbor  algorithm  and  P  held 
constant  at  six. 
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Table  E.7.  Kohonen  Test  Performance  vs  Nodes  with  Six  P-Neighbors 
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For  the  software  parameters,  the  data  seed  matched  the  run  number  and  51  training  vectors 
were  loaded  from  each  class.  The  number  of  nodes  in  layer  0  was  50,  while  the  number  of  nodes  in 
layer  1  is  shown  in  the  table  and  the  number  of  nodes  in  layer  2  was  two.  The  transfer  function 
for  the  nodes  in  layer  1  was  gaussian  while  the  transfer  function  for  the  nodes  in  layer  2  was  linear. 
The  weights  linking  layer  1  nodes  to  the  layer  2  nodes  were  set  using  the  global  MSE  minimization 
algorithm. 

Tables  E.8  and  E.9  show  the  categorization  performance  of  Kernel  Classifier  networks  with 
the  layer  1  weights  trained  using  the  Kohonen  Training  algorithm.  Here  the  spreads  of  the  RBFs 
were  set  using  the  P-Nearest  Neighbors  algorithm  with  P  allowed  to  vary  as  the  square  root  of  the 
number  of  Kohonen  nodes. 


Table  E.8.  Kohonen  Training  Performance  vs  Nodes  with  Variable  P-Neighbors 
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Table  E.9.  Kohonen  Test  Performance  vs  Nodes  with  Variable  P-Neighbors 
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60.30 

3.64 

49 

T 

60.00 

73.00 

79.00 

60.00 

74.00 

67.00 

79.00 

77.00 

63.00 

77.00 

76.70 

3.95 

36 

6 

76.00 

74.00 

61.00 

75.00 

75.00 

63.00 

63.00 

60.00 

60.00 

60.00 

76.60 

3.10 

35 

5 

60.00 

66.00 

60.00 

75.00 

76.00 

77.00 

61.00 

73.00 

61.00 

76.00 

76.60 

4.03 

16 

4 

76.00 

71.00 

77.00 

73.00 

76.00 

76.00 

79.00 

69.00 

76.00 

73.00 

74,90 

3.34 

For  the  software  parameters,  the  data  seed  matched  the  run  number  and  51  training  vectors 
were  loaded  from  each  class.  The  number  of  nodes  in  layer  0  was  50,  while  the  number  of  nodes  in 
layer  1  is  shown  in  the  table  and  the  number  of  nodes  in  layer  2  was  two.  The  transfer  function 
for  the  nodes  in  layer  1  was  gaussian  while  the  transfer  function  for  the  nodes  in  layer  2  was  linear. 
The  weights  linking  layer  1  nodes  to  the  layer  2  nodes  were  set  using  the  global  MSB  minimization 
algorithm. 

E,2,2,S  K~Means  Clustering  Tables  E.IO  and  E.ll  show  the  categorization  perfor¬ 
mance  of  a  Kernel  Classifier  with  the  hidden  layer,  layer  1,  weights  trained  via  the  K-Means 
Clustering  Algorithm  as  the  number  of  nodes  increased. 


Table 

B.IO.  K-Means  Training  Performance  vs  h 

lodes  with  Six  P-Neig 

hbors 

Nodes 

■imm 

Run  1 

■ary 

Run  5 

■imM 

■rrry 

Avf 

■SBHM 

■siHni 

%erct 

Bu3S9 

wnSBm 

%cret 

KiSfiS 

9icict 

9(ctct 

100 

100.00 

100.00 

100.00 

100.00 

99.03 

100.00 

100.00 

100.00 

100.00 

100.00 

90 

100.00 

99.03 

lOO.CO 

100.00 

96.04 

96.04 

100.00 

100.00 

99.03 

100.00 

K  2 

60 

97.06 

99.03 

95.10 

96.04 

97.06 

96.04 

100.00 

100.00 

97.06 

99.03 

70 

94.13 

99.03 

93.16 

96.06 

95.10 

96.04 

100.00 

100.00 

96.06 

96.04 

It 

60 

93.16 

96.04 

93.16 

95.10 

95.10 

97.06 

96.06 

97.06 

94.13 

1.71 

50 

94.13 

96.04 

94.16 

95.10 

94.13 

94.13 

93.14 

94.13 

91.16 

94.13 

1.75 

40 

91.16 

96.06 

66.34 

93.14 

94.13 

94.13 

94.13 

93.14 

90.30 

93.04  - 

3.13 

30 

67.35 

93.16 

67.35 

69.33 

93.16 

90.30 

69.33 

66.34 

69.33 

90.01 

1.57 

30 

86.37 

64.31 

64.31 

86.37 

66.34 

84.31 

76.47 

86.34 

64.31 

83.33 

84.70 

3.33 

10 

70.59 

77.45 

77.45 

84.31 

80.39 

79.41 

77.45 

64.31 

73.55 

63.35 

60.19 

3.96 

Table  E.ll,  K-Means  Test  Performance  vs  Nodes  with  Six  P-Neighbors 


Nodes 

Run  0 
Secret 

Run  1 
%ctci 

Run  3 
9lctct 

Run  3 
Kcict 

Run  4 
%crci 

■iTTlTM 

Bu3±89 

■;Try 

■jTTQ 

BSSSli 

Run  9 
9icret 

Av« 

9lerct 

■BOH 

100 

63.00 

79.00 

74.00 

91.00 

64.00 

66.00 

91.00 

67.00 

76.00 

67.00 

63.90 

mjai 

90 

66.00 

79.00 

73.00 

69.00 

83.00 

65.00 

90.00 

65.00 

76.00 

64.00 

63.70 

80 

60.00 

76.00 

73.00 

60.00 

61.00 

64.00 

67.00 

63.00 

61.00 

66.00 

63.30 

70 

76.00 

60.00 

79.00 

63.00 

76.00 

66.00 

65.00 

65.00 

64.00 

64.00 

63.40 

60 

75,00 

63.00 

75.00 

63.00 

78.00 

60.00 

65.00 

65.00 

64.00 

76.00 

80.90 

3.36 

50 

79.00 

-  79.00 

73.00 

63.00 

76.00 

60.00 

60.00 

60.00 

67.00 

76.00 

79.00 

3.71 

40 

79.00 

74.00 

70.00 

60.00 

79.00 

63.00 

63.00 

60.00 

66.00 

74.00 

78.60 

4.45 

30 

63.00 

71.00 

73.00 

79.00 

79.00 

61.00 

54.00 

77.00 

76.00 

73.00 

74.30 

Hi"  V 

30 

79.00 

70.00 

71.00 

77.00 

73.00 

60.00 

76.00 

73.00 

76.00 

75.00 

75.50 

K  9 

10 

77.00 

-  70.00 

76.00 

73.00 

73.00 

74.00 

79.00 

75.00 

80.00 

73.00 

75.30 

For  the  software  parameters,  the  RBF  sigmas  were  set  via  the  P-Nearest  Neighbor  algorithm 
with  P  held  at  six.  The  data  seed  matched  the  run  number  and  the  data  was  loaded  by  classes. 
The  number  of  nodes  in  layer  0  was  50,  while  the  number  of  nodes  in  layer  1  is  shown  in  the  table 
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Table  E.9.  Kohonen  Test  Performance  vs  Nodes  with  Variable  P-Neighbors 


Nodci 

II 

Rmii  1 
Kctci 

Rva  3 
Kcrct 

Rnm  3 
%<rei 

Rmh  4 

9(ctct 

Rva  5 

Kcret 

Rua  0 
%<tei 

Krnjn 

BrSS!l 

■am 

R33SI 

■aicDi 

AVI 

%€rct 

Std 

%ctc\ 

100 

75.00 

60.00 

00.00 

50.00 

~  50.00 

01.00 

50.00 

56.00 

46.00 

53.00 

50.70 

7.36 

*1 

■1 

00.00 

05.00 

75.00 

70.00 

77.00 

73.00 

04.00 

77.00 

03.00 

77.00 

76.50 

5.16 

04 

mm 

79.00 

70.00 

77.00 

77.00 

79.00 

79.00 

03.00 

90.00 

01.00 

01.00 

00.30 

3.04 

49 

mM 

00.00 

73.00 

79.00 

00.00 

74.00 

07.00 

79.00 

77.00 

03.00 

77.00 

70,70 

3.95 

3« 

70.00 

74.00 

01.00 

75.00 

75.00 

03.00 

03.00 

00.00 

00.00 

00.00 

70.00 

3.10 

35 

00.00 

00.00 

00.00 

75.00 

70.00 

77.00 

01.00 

73.00 

01.00 

76.00 

76.00 

4.03 

10 

n 

70.00 

71.00 

77.00 

73.00 

70.00 

70.00 

-  79.00 

69.00 

76.00 

73.00 

74.90 

3.34 

For  the  software  parameters,  the  data  seed  matched  the  run  number  and  51  training  vectors 
were  loaded  from  each  class.  The  number  of  nodes  in  layer  0  was  50,  while  the  number  of  nodes  in 
layer  1  is  shown  in  the  table  and  the  number  of  nodes  in  layer  2  was  two.  The  transfer  function 
for  the  nodes  in  layer  1  was  gaussian  while  the  transfer  function  for  the  nodes  in  layer  2  was  linear. 
The  weights  linking  layer  1  nodes  to  the  layer  2  nodes  were  set  using  the  global  MSB  minimization 
algorithm. 

E,2,2,S  K^Mtans  Clusitring  Tables  E,  10  and  E.ll  show  the  categorization  perfor¬ 
mance  of  a  Kernel  Classifier  with  the  hidden  layer,  layer  1,  weights  trained  via  the  K-Means 
Clustering  Algorithm  as  the  number  of  nodes  increased. 


Table ! 

E.IO.  K-Means  Training  Performance  vs  N 

fodes  with  Six  P-Neig 

;hbors 

Nodci 

Run  3 

Run  4 

Run  5 

■  R  1  M 

■tnn  ■ 

Rua  9 

AVI 

KZEB 

%<tCX 

%cr€l 

%crct 

9Ccici 

KuSSI 

%CfCt 

96cret 

100 

100.00 

100.00 

100.00 

100.00 

99.03 

100.00 

100.00 

100.00 

100.00 

100.00 

99.90 

0.39 

90 

100.00 

99.03 

lOO.CO 

100.00 

96.04 

90.04 

100.00 

100.00 

99.03 

100.00 

99.41 

0.70 

00 

99.03 

95.10 

1 

97.06 

90.04 

100.00 

100.00 

97.06 

99.03 

90.33 

1.53 

70 

99.03 

93.16 

!  ^B 

95.10 

90.04 

100.00 

100.00 

96.00 

90.04 

97.16 

3.30 

60 

90.04 

KT  r  1^1 

'  ^B 

95.10 

97.06 

96.00 

97.06 

97.06 

94.13 

95.59 

1.71 

50 

94.13 

90.04 

r 

94.13 

94.13 

93.14 

95.10 

94.13 

91.10 

94.13 

1.75 

40 

91.10 

96.00 

^R  '  ^B 

94.13 

94.13 

94.13 

93.14 

93.14 

90.30 

93.04 

3.13 

30 

07.35 

93.16 

09.33 

93.16 

90.30 

59.33 

91.30 

00.34 

09.33 

90.01 

1.57 

30 

06.37 

04.31 

04.31 

06.37 

00.34 

04.31 

76.47 

60.34 

04.31 

03.33 

04.70 

3.33 

10 

70.59 

77.45 

77.45 

04.31 

00.39 

79.41 

77.45 

04.31 

73.55 

03.35 

00.19 

3.96 

Table  E.ll.  K-Means  Test  Performance  vs  Nodes  tvith  Six  P-Neighbors 


iHim 

Run  1 
r»ctet 

■ansi 

IB39SBS 

K3SBI 

Run  4 
Keret 

■3SB9 

Run  7 
%ctci 

■axn 

BOSS31 

Run  9 
%ctci  - 

%CICi 

100 

03.00 

79.00 

74.00 

91.00 

04.00 

06.00 

91.00 

07.00 

76.00 

07.00 

03.90 

5.34 

90 

06.00 

79.00 

73.00 

09.00 

03.00 

05.00 

90.00 

05.00 

70.00 

04.00 

03.70 

4.06 

60 

OO.'^O 

76.00 

73.00 

06.00 

01.00 

04.00 

07.00 

03.00 

01.00 

06.00 

03.30 

KlyH 

70 

70.00 

00.00 

Kiiia 

03.00 

70.00 

06.00 

-  05.00 

05.00 

64.00 

04.00 

63.40 

VI  nS 

60 

75.00 

03.00 

K.'  iS 

03.00 

76.00 

00.00 

65.00 

05.00 

04.00 

70.00 

00.90 

VI' V 

79.00 

79.00 

■  is 

03.00 

70.00 

60.00 

00.00 

00.00 

07.00 

76.00 

79.00 

3,71 

79.00 

74.00 

■  !  •!■ 

60.00 

79.00 

03.00 

03.00 

00.00 

66.00 

74.00 

76.60 

4.45 

03.00 

71.00 

B:  im 

79.00 

79.00 

01.00 

54.00 

77.00 

70.00 

73.00 

74.30 

VTT  V 

79.00 

70.00 

71.00 

77.00 

73.00 

00.00 

70.00 

73.00 

76.00 

75.00 

75.50 

Vf^  V 

77.00 

70  00 

70.00 

73.00 

73.00 

74.00 

79.00 

75.00 

00  00 

73  00 

75.30 

BESB 

For  the  software  parameters,  the  RBF  sigmas  were  set  via  the  P-Nearest  Neighbor  algorithm 
with  P  held  at  six.  The  data  seed  matched  the  run  number  and  the  data  was  loaded  by  classes. 
The  number  of  nodes  in  layer  0  was  50,  while  the  number  of  nodes  in  layer  1  is  shown  in  the  table 
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Center  at  Class-Clusier  Averages  Tables  E.14  and  E.15  show  the  categoriza¬ 
tion  performance  for  a  Kernel  Classifier  network  whose  hidden  layer  weights  were  trained  using  the 
Center  at  Class-Cluster  Averages  algorithm.  Table  E.16  shows  the  number  of  layer  1  nodes  created 
to  cover  the  input  data  space  for  each  run. 


Table  E.14.  Center  at  Class  Averages  gaining  Performance  vs  Avg  Threshold 


Rum  0 
Wcrct 

Rum  1 
%ct«t 

Wiiiy 

■SSfiB 

Rum  4 

%ctcl 

WtH 

Rum  5 

%erct 

Rum  7 
licrct 

■fcniM 

■289 

Rum  9 
Kcret 

Avg 

lictcl 

Kctcl 

0.35 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

lOO.OO 

100.00 

100.00 

100.00 

100.00 

0.00 

O.SO 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

0.00 

0.75 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

0.00 

1.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

0.00 

1.35 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

0.00 

1.50 

90.03 

95.04 

95.04 

95.04 

100.00 

99.03 

100.00 

100.00 

100.00 

99.13 

0.51 

1.75 

95.04 

95.05 

95.05 

97.05 

96.05 

97.05 

95.04 

99.03 

100.00 

97.55 

1.53 

95.05 

94.13 

94.13 

95.14 

95.10 

95.14 

95.05 

97.05 

100.00 

95.59 

1.95 

91.15 

59.33 

59.33 

94.15 

93.15 

95.14 

95.10 

95.05 

93.17 

3.35 

90.03 

54.50 

54.50 

57.35 

95.14 

94.13 

91.30 

90.30 

59.49 

5.11 

93.15 

91.15 

75.47 

75.47 

57.55 

93.15 

54.51 

53.55 

54.51 

KS9 

55.45 

5.09 

5.00 

57.35 

74.51 

74.51 

54.51 

57.35 

55.53 

77,45 

55.39 

55.55 

53.45 

4.77 

5.35 

57.35 

53.75 

53.75 

50.59 

75.50 

59.51 

55.57 

77.45 

51.57 

75.47 

5.91 

5.50 

70.55 

50.75 

50.75 

75  49 

59.51  i 

55.55 

50.95 

53.75 

55.75 

9.55 

5.75 

55.55 

55.53 

55.53 

51,75 

55.&5 

59.50 

55.55 

51.95 

53.75 

55.35 

9.33 

4.00 

51.75 

50.00 

50.00 

55.55 

55.65 

50.00 

55.55 

51.95 

55.53 

54.50 

5.91 

Table  E.15.  Center  at  Class  Averages  vs  Avg  Threshold 


K3S9 

mmwm 

1289 

■289 

■jVrni 

■2389 

Run  4 
%ctci 

Rum  5 

%CtCt 

nftum  5 
%crct 

Run  7 
Tlcfcl 

■3999 

■SB9 

Avf 

%ctc4 

1281 

0.35 

55.00 

51.00 

77.00 

57.00 

53.00 

57.00 

55.00 

55.00 

55.00 

55.00 

84.00 

0.50 

55.00 

53.00 

77.00 

57.00 

53.00 

57.00 

59.00 

55.00 

55.00 

54.00 

54.10 

0.75 

51.00 

51.00 

77.00 

55.00 

55.00 

55.00 

59.00 

55.00 

53.00 

54.00 

53.50 

KSB 

1.00 

53.00 

51.00 

77.00 

54.00 

55.00 

55.00 

55.00 

54.00 

55.00 

54.00 

53.40 

3.54 

1.35 

53.00 

50.00 

77.00 

55.00 

54.00 

55.00 

56.00 

55.00 

53.00 

55.00 

53.40 

3.54 

1.50 

55.00 

51.00 

75.00 

55.00 

55.00 

90.00 

55.00 

54.00 

53.00 

54.00 

53.90 

3.95 

1.75 

55.00 

79.00 

50.00 

50.00 

53.00 

90.00 

54.00 

55.00 

53.00 

53.00 

53.70 

3.00 

54.00 

75.00 

55.00 

77.00 

75.00 

51.00 

55.00 

53.00 

75.00 

50.00 

50.40 

3.17 

55.00 

74.00 

79.00 

79.00 

50.00 

55.00 

55.00 

53.00 

79.00 

50.00 

50.70 

3.39 

53.00 

75.00 

50.00 

74.00 

75.00 

55.00 

55.00 

75.00 

75.00 

75.00 

79.50 

4.37 

50.00 

71.00 

75.00 

75.00 

77,00 

53.00 

55.00 

71.00 

65.00 

75.00 

75.50 

5.03 

5.00 

75.00 

73.00 

75.00 

55.00 

55.00 

75.00 

53.00 

59.00 

51.00 

73.00 

75.40 

5.57 

5.35 

77.00 

73.00 

57.00 

57.00 

75.00 

74.00 

71.00 

64.00 

55.00 

70.00 

73.10 

5.35 

5.50 

75.00 

75.00 

53.00 

59.00 

59.00 

71.00 

71.00 

57.00 

57.00 

54.00 

65.50 

5.91 

5.75 

75.00 

69.00 

50.00 

59.00 

54.00 

60.00 

59.00 

55.00  1 

51.00 

54.00 

51.50 

5.05 

4.00 

57.00 

55.00 

59.00 

50.00 

53.00 

59.00 

50.00 

55.00  1 

51.00 

55.00 

54.70 

3.47 

Table  E.16.  Nodes  Generated  for  Center  at  Class  Averages  vs  Avg  Threshold 


Aver*g« 

Thtethold 

Run  0 
Node» 

Run  1 
Nodes- 

Run  3 
Nodes 

Run  3 
Nodes 

Run  4 
Nodes 

Rua  5 
Nodes 

Run  6 
Nodes 

Run  7 
Nodes 

Run  5 
Nodes 

Run  9 
Nodes 

Avg 

Nodes 

Std 

Nodes 

0.35 

103.00 

200.00 

201.00 

203.00 

99.00 

200.00 

99.00 

200.00 

100.00 

200.30 

■Sii 

0.50 

95.00 

59.00 

93.00 

96.00 

93.00 

95.00 

90.00 

91.00 

59.00 

93.10 

55.00 

77.00 

55.00 

56.00 

54.00 

55.00 

53.00 

51.00 

■T~?'^S 

51.00 

IT  iS 

77.00 

73.00 

75.00 

75.00 

76.00 

75.00 

75.00 

69.00 

75.00 

KiLM 

55.00 

64.00 

65.00 

56,00 

63.00 

65.00 

66.00 

61.00 

■T~7'iS 

66.00 

3.35 

1.50 

54.00 

45.00 

54.00 

50.00 

50.00 

53.00 

54.00 

51.00 

55.00 

3.53 

1.75 

43.00 

36.00 

41.00 

40.00 

41.00 

41.00 

39.00 

37.00 

40.00 

43.00 

3.05 

3.00 

33.00 

35.00 

33.00 

35.00 

35.00 

33.00 

33.00 

30.00 

39.00 

35.00 

30.60 

3.34 

3.35 

34.00 

31.00 

34.00 

30.00 

34.00 

33.00 

37.00 

34.00 

33.00 

33.00 

33.70 

3.07 

3.50 

19.00 

19.00 

15.00 

15.00 

15.00 

16.00 

30.00 

15.00 

14.00 

15.00 

17.50 

3.56 

3.75 

14.00 

15.00 

14.00 

11.00 

14.00 

13.00 

15.00 

13.00 

11.00 

13.00 

13.70 

3.45 

3.00 

13.00 

13.00 

13.00 

13.00 

13.00 

13.00 

10.00 

10.00 

11.00 

11.60 

1.55 

3.35 

11.00 

13.00 

10.00 

13.00 

9.00 

5.00 

5.00 

5.00 

5.00 

9.50 

3.30 

3.50 

9.00 

9.00 

9.00 

7.00 

7.00 

5.00 

4.00 

4.00 

6.00 

6.50 

1.59 

3.75 

5.00 

5.00 

7.00 

4.00 

5.00 

6.00 

3.00 

3.00 

6.00 

5.50 

3.06 

4  00 

3  00 

500 

6  CO 

HJIszH 

3  CO 

4  CO 

—  CO 

SCO 

3  CO 

5  CO 

5  70 

1  19 

For  the  software  parameters,  the  average  threshold  was  allowed  to  vary,  the  data  seed  matched 
the  run  number,  the  sigma  threshold  was  set  at  four  and  the  training  rule  for  the  RBF  sigma  was 
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the  Scale  Sigma  According  to  Class  Interference  with  interference  threshold  of  .4.  The  data  seed 
matched  the  run  number  while  51  training  vectors  were  loaded  for  each  class.  The  number  of  nodes 
in  layer  0  was  50,  while  the  number  of  nodes  in  layer  1  is  shown  in  the  table  and  the  number  of 
nodes  in  layer  2  was  two.  The  transfer  function  for  the  nodes  in  layer  1  was  gaussian  while  the 
transfer  function  for  the  nodes  in  layer  2  was  linear.  The  weights  linking  layer  1  nodes  to  layer  2 
nodes  were  set  using  the  global  MSE  minimization  algorithm. 

E,2.2,5  PNN/RBF  Comparison  Tables  E.17  and  E.18  show  the  categorization  per¬ 
formance  of  a  PNN  as  the  sigma  varied.  The  weights  in  the  hidden  layer  were  trained  using  the 
Nodes  at  Data  Points  algorithm. 


Table  E.17.  PNN  Training  Performance  vs  Sigma 


Bun  1 
%<tc% 
100.00 
100.00 
94.10 


1.3S 

1.30 


Bub  0 
%ctct 

Std 

%«ct 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00, 

0.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

0.00 

97.00 

97.00 

90.06 

90.00 

95.10 

97.00 

94.19 

95.10 

98.00 

95.79 

1.00 

05.99 

90.90 

91.10 

89.99 

91.10 

07.95 

08.97 

05.55 

87.94 

9.45 

70.47 

75.49 

77.45 

70.45 

79.49 

77.45 

77.45 

79.55 

78.77 

9.40 

09.75 

00.05 

09.75 

09.01 

00.70 

00.70 

ml%lW 

70.59 

81.78 

84.81 

5.01 

55.00 

57.09 

50.09 

00.70 

54.91 

51.90 

55.99 

59.00 

57.04 

58.07 

9.59 

:  50.00 

55.99 

51.90 

50.00 

59.94 

50.90 

50.00 

54.10 

55.99 

59.57 

9.00 

50.00 

50.90 

50.90 

54.91 

50.00 

50.00 

50.00 

51.98 

50.00 

50.90 

1.48 

50.00 

50.00 

50.90 

59.94 

50.00 

50.00 

50.00 

50.00 

50.00 

50.59 

0.90 

50.00 

60.00 

50.00 

59.94 

50.00 

50.00 

50.00 

50.00 

50.00 

50.99 

0.80 

50.00 

50.00 

50.00 

50.90 

50.00 

50.00 

50.00 

50.00 

50.00 

50.10 

0.99 

For  the  software  parameters,  the  output  threshold  was  set  to  one.  The  training  vectors  were 
loaded  by  class  with  51  vectors  from  each  class  and  a  data  seed  of  six.  The  number  of  nodes  in 
layer  0  was  50,  while  the  number  of  nodes  in  layer  1  was  102  and  the  number  of  nodes  in  layer  2 
was  two.  The  transfer  function  for  the  nodes  in  layer  1  was  gaussian  while  the  transfer  function 
for  the  nr^des  in  layer  2  was  linear.  The  weights  linking  layer  1  nodes  to  the  layer  2  nodes  v/ere  set 
using  the  PNN  algorithm. 

Tables  E.19  and  E.20  show  the  categorization  performance  of  an  RBF  Kernel  Classifier  net¬ 
work  as  the  sigma,  or  spread,  varied.  The  weights  in  the  hidden  layer  were  trained  using  the  Nodes 
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at  Data  Points  algorithm. 


Tab! 

e  E.19. 

RBF] 

'^etwor! 

<  Training  Performance  vs  Sigma 

Illljlll 

lUfTTVV 

KSSfll 

HJiQinB 

mSSSM 

WtSBBm 

Avg 

Kctct 

0.35 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

0.00 

0.50 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

0.00 

0.75 

100.00 

100.00 

100.00 

100.00 

98.04 

100.00 

100.00 

100.00 

100.00 

98.04 

99.61 

0.78 

1.00 

100.00 

01.18 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

100.00 

99.13 

3.65 

1.35 

100.00 

100.00 

100.00 

100.00 

96.08 

100.00 

100.00 

100.00 

100.00 

100.00 

99.61 

1.18 

1.50 

100.00 

100.00 

100.00 

100.00 

87.35 

100.00 

100.00 

100.00 

100.00 

94.13 

98.14 

4.05 

1.75 

100.00 

60.78 

100.00 

91.18 

100.00 

85.55 

56.86 

100.00 

98.04 

100.00 

86.57 

18.67 

3.00 

83.55 

100.00 

100.00 

57.84 

48.09 

75.49 

87.35 

53.94 

50.98 

81.57 

74.81 

19.50 

3.35 

50.00 

50.00 

97.06 

61.76 

50.00 

50.00 

60.78 

57.84 

53.94 

100.00 

64.13 

17.78 

3.50 

55.03 

50.00 

50.00 

50.98 

50.00 

50.00 

50.00 

50.98 

54.90 

50.00 

51.08 

1.73 

3.75 

50,00 

5000 

50.00 

50.98 

50.00 

50.00 

50.00 

50.00 

51.96 

50.00 

50.39 

0.65 

5.00 

50.00 

50.00 

50.00 

50.00 

50.98 

50.00 

50,98 

70.59 

50,98 

53.65 

6.09 

Table  E.20.  RBF  Network  Test  Performance  vs  Signia 


Sigtn4 

K2SI 

Run  3 
%erct 

ISSBB 

Run  4 
%ctc\ 

Run  5 
9(erct 

hun  6 
Kcrct 

Run  7 
9;erct 

Run  6 
9lctct 

Run  9 
9icict 

Run  10 

%CICt 

Avr 

9;cict 

1^1 

0.35 

74.00 

73.00 

83.00 

77.00 

79.00 

84.00 

64.00 

85.00 

86.00 

86.00 

80.90 

4.85 

0.50 

78.00 

73.00 

83.00 

79.00 

80.00 

84.00 

88.00 

85.00 

88.00 

88.00 

63.50 

4.78 

83.00 

73.00 

85.00 

79.00 

76.00 

87.00 

86.00 

86.00 

86.00 

91.00 

63.50 

5.40 

■VS 

83.00 

70.00 

89.00 

80.00 

80.00 

89.00 

87.00 

83.00 

85.00 

89.00 

63.40 

5.61 

83.00 

74.00 

90.00 

80.00 

77.00 

87.00 

88.00 

81.00 

84,00 

87.00 

85.00 

4.88 

1.50 

83.00 

74.00 

90.00 

79.00 

69.00 

87.00 

90.00 

81.00 

84.00 

03.:o 

6.39 

1.75 

79.00 

47.00 

89.00 

67.00 

80.00 

69.00 

53.00 

83.00 

87,00 

75.60 

3.00 

70.00 

71.00 

89.00 

51.00 

49,00 

69.00 

79.00 

50.00 

57.00 

63.50 

3.35 

50.00 

50.00 

89.00 

61.00 

50.00 

50.00 

58.00 

44.00 

86.00 

59.00 

■xn 

3.50 

50.00 

45.00 

50.00 

53.00 

50,00 

50.00 

46.00 

55.00 

59,00 

51.40 

5.90 

3.75 

50.00 

49.00 

50.00 

50.00 

50.00 

50.00 

50.00 

50.00 

50.00 

50.00 

49,90 

0.30 

3.00 

50.00 

53.00 

50.00 

50.00 

50.00 

57.00 

47.00 

53.00 

64.00 

45.00 

51.80 

5.10 

For  the  software,  parameters,  the  output  threshold  was  set  at  one.  The  training  vectors  were 
loaded  by  class  with  51  vectors  from  each  class  and  a  data  seed  of  six.  The  number  of  nodes  in 
layer  0  was  50,  while  the  number  of  nodes  in  layer  1  was  102  and  the  number  of  nodes  in  layer  2 
was  two.  The  transfer  function  for  the  nodes  in  layer  1  was  gaussian  while  the  transfer  function 
for  the  nodes  in  layer  2  was  linear.  The  weights  linking  layer  1  nodes  to  the  layer  2  nodes  were  set 
using  the  global  MSB  minimization  algorithm. 


E,8  Radar  System  Characierizaiion 

This  data  consisted  of  300  training  and  1990,  6  dimension  pattern  vectors.  These  pattern 
vectors  represented  one  of  ten  radar  platforms.  Only  Kernel  Classifier  networks  were  developed 
to  categorize  this  data.  The  networks  consisted  of  a  standard  RBF  network  and  an  RBF-based 
arbitrator  network. 

E,8J  RBF  Network  The  topology  for  this  network  is  shown  in  figure  3.7.  Table  E.21  shows 
the  categorization  performance  of  the  RBF  network  trained  to  categorize  the  ten  radar  platforms. 
This  network  consisted  of  6  nodes  in  layer  0,  and  155  nodes  in  layer  1,  and  10  nodes  in  layer  2. 
For  the  software  parameters,  the  transfer  function  for  each  node  in  layer  0  was  the  identity  transfer 
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Tabl_e_E_._21^RBE_Network  Performance 


Classification 

Threshold 

%  Correct 

Training 

Test 

.8 

87.00 

72.29 

.6 

89.67 

77.91 

.4 

93.67 

82.88 

.2 

93.67 

82.88 

li _ -2. _ 

93.67 

82;88 

T:^ble^._22:._Network  AJgerfoLmance 


Classification 

Threshold 

%  Correct  | 

Training 

Test  II 

.8 

.6 

.4 

95.00 

iXi 

.2 

95.00 

90.05 

.0 

95.00 

90.05 

function,  while  the  transfer  function  for  each  node  in  layer  1  was  gaussian  and  layer  2  was  linear. 
The  weights  for  the  layer  1  nodes  were  set  via  the  Center  at  Class-Cluster  Averages  algorithm  with 
the  average  threshold  set  at  .03.  The  spreads  were  set  using  the  Scale  Sigmas  According  to  Class 
Interference  algorithm  with  the  interference  threshold  set  at  .5.  The  weights  linking  the  layer  1 
nodes  to  the  layer  2  nodes  were  set  via  global  minization  of  the  MSE. 

E.S,2  Arbiiraior  The  topology  for  this  network  is  shown  in  figure  5.15.  Table  E.22  shows 
the  performance  of  Network  A  trained  to  categorize  five  radar  platforms.  This  network  consisted 
of  6  nodes  in  layer  0,  and  175  nodes  in  layer  1,  and  6  nodes  in  layer  2.  For  the  software  parameters, 
the  transfer  function  for  each  node  in  layer  0  was  the  identity  transfer  function,  while  the  transfer 
function  for  each  node  in  layer  1  was  gaussian  and  layer  2  was  linear.  The  weights  for  the  layer  1 
nodes  were  set  via  the  Center  at  Class-Cluster  Averages  algorithm  with  the  average  threshold  set 
at  .03.  The  spreads  were  set  using  the  Scale  Sigmas  According  to  Class  Interference  algorithm  with 
the  interference  threshold  set  at  .5.  The  weights  linking  the  layer  1  nodes  to  the  layer  2  nodes  were 
set  via  global  minization  of  the  MSE. 

Tabic  E.23  shows  the  performance  of  Network  B  trained  to  categorize  five  radar  platforms. 
This  network  consisted  of  6  nodes  in  layer  0,  and  173  nodes  in  layer  1,  and  5  nodes  in  layer  2.  For 
the  software  parameters,  the  transfer  function  for  each  node  in  layer  0  was  the  identity  transfer 
function,  while  the  transfer  function  for  each  node  in  layer  1  was  gaussian  and  layer  2  was  linear. 
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Tab]e_E,23».Network  B  Eerformance 


deification 

Threshold 

%  Correct 

Training 

Test 

.8 

78.33 

.6 

95.67 

88.31 

.4 

98.67 

92.64 

.2 

98.67 

92.64 

.0 

98.67 

92.64 

The  weights  for  the  layer  1  nodes  were  set  via  the  Center  at  Class-Cluster  Averages  algorithm  with 
the  average  threshold  set  at  .01,  The  spreads  were  set  using  the  P-Neighbors  algorithm  with  the 
number  of  neighbors  set  at  .5.  The  weights  linking  the  layer  1  nodes  to  the  layer  2  nodes  were  set 
via  global  minization  of  the  MSE. 

Table  E.24  shows  the  performance  of  the  total  Arbitrator  network  trained  to  categorize  the 
ten  radar  platforms.  This  network  consisted  of  6  nodes  in  layer  0  feeding  networks  A  and  B.  The 


Table  E.24.  Arbitration  Network  Performance 


Classification 

Threshold 

%  Correct 

Training 

Test  II 

.8 

.6 

98.33 

81.43 

.4 

99.33 

85.79 

.2 

99.33 

86.35 

99.33 

86.35 

outputs  of  these  networks  were  passed  into  Network  C  which  consisted  of  10  nodes  in  its  input 
layer,  75  RBF  nodes  and  10  output  nodes.  For  the  software  parameters,  the  transfer  function  for 
each  RBF  node  was  gaussian  and  each  output  node  was  linear.  The  weights  for  the  RBF  nodes 
were  set  via  the  Center  at  Class-Cluster  Averages  algorithm  with  the  average  threshold  set  at  .03. 
The  spreads  were  set  using  the  Scale  Sigmas  According  to  Class  Interference  algorithm  with  the 
interference  threshold  set  at  .5.  The  weights  linking  the  RBF  nodes  to  the  output  nodes  were  set 
via  global  minization  of  the  MSE. 
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Appendix  F.  Software  Analysis 


FJ  Iniroduciion 

The  software  to  be  described  in  this  chapter  was  designed  according  to  the  object-oriented 
approach  presented  in  Chapter  4.  This  chapter  begins  with  a  description  of  the  data  structures 
implemented  for  the  software  and  concludes  with  a  discussion  of  the  software  modules  and  the 
mapping  of  the  training  algorithms,  developed  in  chapter  3,  into  software  functions. 

Object  Oriented  Structure 

The  structure  of  the  software  centered  on  the  attributes  of  the  nodes*  weights,  sigmas,  con¬ 
nections,  transfer  function  and  class  listed  in  Chapter  4. 

F»2J  Weights  A  given  node*s  weights  are  defined  as  the  factor  by  which  an  output  signal 
from  another  node  is  multiplied  before  being  processed  by  that  node.  For  example,  as  shown  in 
figure  F.l,  node[2]-weight[l]  would  be  the  weight  which  multiplies  the  output  of  node  1  prior  to 
entering  node  2  for  processing.  Conversely,  node[l]-weight[2]  would  be  the  weight  which  multiplies 
the  output  of  node  2  prior  to  entering  node  1  for  processing.  Finally,  node[l]-weight[l]  would  be 
the  amplification  factor  for  the  output  of  a  node[l]  feeding  back  into  node[l). 

F,2,2  Sigmas  A  given  node*s  sigmas  are  defined  as  the  offset  factor  for  a  node  with  a 
sigmoidal  transfer  function  or,  conversely,  the  spread  factor,  in  a  particular  direction,  for  the 
gaussian  transfer  function.  This  factor  thus  controls  a  node*s  response  to  a  given  input.  For 
example,  as  shown  in  figure  F.2,  node[2]-sigma[l]  would  be  the  sigma  factor  for  a  signal  passing 
from  the  output  of  node  1  to  the  input  of  node  2.  Conversely,  node[l]-sigma(2]  would  be  the  sigma 
factor  for  a  signal  passing  from  the  output  of  node  2  to  the  input  of  node  1.  Finally,  node-record  [1]- 
sigma[l)  would  be  internal  sigma  for  node  1. 

F,2.S  Connect  A  node*s  connections  to  other  nodes  in  the  network  are  defined  as  the  con¬ 
nection  by  which  an  output  signal  from  another  node  is  allowed  to  pass  to  that  node.  For  example, 
as  shown  in  figure  F.3,  node[2)-connect[l)  would  be  set  to  a  1  if  node  2  received  output  from  node 
1.  Conversely,  if  node  2  did  not  receive  node  l*s  output,  node[2]-connect[l]  would  be  set  to  a  0. 
Similarly,  node[l]-connect[2]  would  be  1  if  node  1  received  node  2*s  output  and  0  if  node  1  did  not 
receive  node  2*s  output.  Finally,  node[l]-connect[l]  will  be  set  to  a  1  if  node  1  to  received  input 
from  itself. 
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Figure  F.l.  Node  Weight  Structure 


node  2 
sigma  2 


Figure  F.2.  Node  Sigma  Structure 


Figure  F.3.  Node  Connection  Structure 


F.2.4  Transfer  Function  A  given  nodes  output  will  be  a  function  of  the  inputs,  weights, 
and  sigmas  connecting  the  node  to  the  other  nodes.  Letting  5,  be  the  input  vector  for  a  given 
node,  w  the  weight  vector,  and  a  be  the  offset  vector  then  the  possible  transfer  functions  will  be 
the  linear  transfer  function  of  the  form: 

f{x)  =  xw  (F.l) 

or  a  sigmoidal  function  of  the  form: 

/(i)  =  [l  +  e-(*-®+‘'‘)]-‘  (F.2) 

or  a  gaussian  function  of  the  form: 

f{x)  =  6"“^^  (F.3) 

or  the  identity  function  of  the  form: 

f(xi)  =  Xi  (F.4) 


However,  any  other  applicable  transfer  function  could  also  be  assigned. 


F.2,5  Class  A  given  node  can  be  assigned  a  class  to  which  the  node  is  responsible  for 
responding.  This  is  applicable  to  nodes  whose  transfer  function  is  the  gaussian  function  as  well  as 
to  output  nodes. 

F.3  Software  Analysis 

The  software  modules  implemented  in  this  theses  are  described  in  detail  in  the  following 
sections.  The  code  for  these  modules  is  found  in  Appendix  G. 

F.SJ  NETMENU  This  module  is  the  overall  controlling  module  of  the  network.  It  provides 
the  user  interface  to  the  software  via  the  SUN  terminal  and  keyboard  and  calls  the  appropriate 
modules  to  execute  the  network  decision.  NETMENU  allows  the  selection  of  the  type  of  network 
to  be  configured,  the  number  of  layers  and  nodes  in  the  network,  and  the  training  rule  of  for 
the  weights  of  each  node.  Currently,  the  allov/ed  training  rules  for  the  weights  in  layer  1  are  to 
train  the  weights  to  match  the  features,  center  the  weights  at  class  averages,  establish  the  weights 
via  the  K-means  algorithm,  establish  the  weights  via  Kohonen  training,  or  train  the  weights  via 
backpropagation  using  the  MSE,  CE  and  CFE  algorithms.  Allowed  training  rules  for  the  weights 
in  layer  2  are  matrix  inversion  and  backpropagation  using  the  MSE,  CE  and  CFE  algorithms  and 
the  PNN  Implementation.  Allowed  training  rules  for  the  weights  in  layer  3  are  backpropagation 
using  the  MSE,  CE  and  CFE  algorithms. 

Once  the  training  rules  are  established,  NETMENU  will  allow  the  transfer  function,  for  the 
nodes  in  each  layer,  to  be  selected.  At  this  time,  the  only  valid  transfer  functions  are  the  identity, 
the  gaussian,  the  sigmoidal,  and  the  linear  transfer  functions.  Also,  each  node  in  the  same  layer 
will  be  assigned  the  same  transfer  function. 

If  any  of  the  layers  are  assigned  the  gaussian  transfer  function,  NETMENU  allows  the  selection 
of  a  training  rule  for  the  thresholds  or  sigmas.  At  this  time,  the  thresholds  may  be  trained  by  setting 
them  to  a  constant,  scaling  the  sigmas  based  on  class  interference,  and  setting  the  sigmas  equal  to 
the  P-Neighbor  average  distances. 

Once  these  parameters  have  been  established,  NETMENU  will  then  preset  the  weights,  sig¬ 
mas,  or  outputs  for  the  network,  if  desired,  or  randonily  initialize  the  weights  and  sigmas.  Any 
preset  data  wilbbe  read  from  input- data  files  .specifying  the  appropriate  reference  number  for  the 
node  and  the  preset  value. 

After  reading  in  both  the  training  and  test  data  files,  NETMENU  will  then  configure  the 
network  by  establishing  the  appropriate  nodal  connections  and  assigning  the  nodes  their  appropriate 
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transfe*:  function. 


From  the  selected  training  rule,  NETMENU  will  then  call  the  appropriate  training  function, 
in  NETTRAIN,  to  train  the  weights  in  each  of  the  layers.  Once  the  network  has  been  fully  trained, 
NETMENU  then  establishes  the  accuracy  of  the  network  by  calling  NETERROR  to  apply  both 
the  test  and  training  data  to  the  network,  calculate  the  network  output,  and  perform  a  percentage 
calculation.  This  percentage  calculation  is  made  for  the  training  data,  test  data  and  total  data.  A 
correct  response  occurs  when  the  appropriate  node  produces  an  output,  above  some  user  predefined 
threshold,  which  is  greater  than  all  the  other  output  nodes. 

F.S.2  NETERROR  This  module  contains  the  functions  necessary  to  determine  the  networks 
classification  of  a  data  vector  and  the  error  performance  of  the  network. 

F,8,2,l  Test  The  Network  This  function  takes  an  unknown  data  vector  from  the  call¬ 
ing  module,  applies  the  data  vector  to  the  network  and  calculates  the  network  output.  If  their 
largest  output  for  any  node  is  below  a  predefined  threshold,  the  unknown  pattern  will  be  assigned 
a  class  of  zero  to  indicate  an  unclassified  pattern.  If  the  largest  output  is  above  the  threshold, 
the  unknown  pattern  will  be  assigned  the  same  classification  as  that  of  the  winning  node.  For 
classifications  in  which  the  network  class  does  not  match  the  input  data  class,  an  error  count  is 
updated. 


F.S.2,2  Determine  Class  as  Largest  This  function  classifies  the  input  data  vector  as 
belonging  to  one  of  the  possible  output  classes  providing  the  highest  output  is  above  some  user 
selected  threshold.  This  allows  the  user  to  determine  the  point  at  which  the  network  is  allowed  to 
consider  an  input  data  pattern  is  classified. 

F.3.2.8  Update  Errors  This  function  increments  an  error  count  for  each  time  the  net¬ 
work  classification  does  not  match  the  proper  classification  of  the  data. 

F.8.8  NETTRAIN  NETTRAIN  contains  the  submodules  necessary  to  establish  the  network 
weights  via  the  following  training  procedures: 

a)  Global  MSE  Minimization 

b)  Make  Nodes  at  Data  Points 

c)  Center  Weights  at  Class  Averages 

d)  K-means  Cluster 
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e)  Train  via  Kohoncn 

f)  MSE  Remaining  Layers 

g)  CE  Remaining  Layers 

h)  CFM  Remaining  Layers 

i)  PNN  Last  Layer 

j)  Scale  Sigma  by  Class  Interference 

k)  Set  Sigmas  to  Constant 

l)  Set  Sigma  at  P  Neighbor  Avg 

Each  of  these  submodules  accomplishes  its  training  routine  by  executing  the  specialized  functions 
contained  in  NETAUX, 

F.S.SJ  Global  MSE  Minimization  Global  MSE  Minimization  performs  the  optimiza¬ 
tion  of  the  weights  linking  the  radial  basis  function  nodes  in  the  hidden  layer  to  the  nodes  in  the 
output  layer  using  the  equations  established  in  section  3.5.  This  submodule  first  calls  the  function 
"Determine  Y  Matrix".  This  function  applies  each  data  record  in  the  training  set  and  calculates 
the  output  for  each  of  the  L  nodes  in  layer  1.  That  is,  the  output  for  each  of  the  nodes  due  to  the 
pattern  will  be 


Voui  —  [ypl )  yp2i  VpZy  •  •  • )  2/pJs] 


(F.5) 


This  process  is  accomplished  for  each  of  the  P 


patterns  in  the  training  set  to  produce  the  matrix: 


y  = 


2/11 

yi2 

•  •  •  yiL 

2/21 

J/22 

•  •  •  y2L 

yp\  yp2  .  •  •  ypL 


(F.6) 


Global  MSE  Minimization  then  computes  the  S  matrix  by  calling  the  function  "Determine  S  Matrix” 
to  determine  the  required  outputs  for  each  of  the  M  nodes  in  the  output  layer  due  to  an  input 
pattern.  This  function  tests  the  classification  assigned  to  a  data  record.  If  the  class  of  the  data 
record  p  is  Ny  then  the  output  of  the  (i\^  —  1)*^  node  in  the  output  layer  is  assigned  a  value  of  1 
while  all  other  nodes  are  assigned  a  value  of  zero: 
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Sp  =  [rfpi,  dp2(  •  •  • )  dpN-i,dppr,  dpN+i . dpM] 


(F.7) 


which  translates  to 


sp  =  [0,0 . 1,0,0,...,0] 


(F.8) 


This  process  is  accomplished  for  each  of  the  P  records  in  the  training  set  until  a  P  by  M  matrix 
is  established  where  each  row  contains  the  desired  output  for  the  last  layer  nodes  due  to 
input  pattern. 


5  = 


<^11 

di2 

...  diM 

dai 

d22 

•  • .  diM 

dpi 

dpi 

. . .  dpL 

(F.9) 


’’Global  MSB  Minimization”  then  calls  the  function  "Determine  M  Matrix”  to  compute  the  sum¬ 
mation  of  the  product  of  the  output  of  a  node  in  the  first  layer  with  the  output  of  a  node 
in  the  first  layer  over  all  P  patterns.  That  is 

Mib  =  22p=i  yplVpB  as  defined  in  equation  C.61.  Assuming  there  are  L  nodes  in  the  layer, 
the  M  matrix  then  becomes: 


M  = 


Mn 

Mi2 

...  Mil 

Mil 

M22 

. . .  Mil 

Mil 

Ml2 

. . .  Mll 

(F.IO) 


After  executing  the  function  ’’Make  Identity  Matrix”  to  form  an  L  by  L  identity  matrix  as: 


N  = 


1  0  ...  0 

0  1  ...  0 

0  0  ...  1 


(F.ll) 
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”  Global  MSE  Minimization”  then  calls  the  function  ”  Determine  Matrix  Transpose”  which  takes 
the  M  matrix  and  returns  its  transpose  back  as  a  new  M  matrix.  "Global  MSE  Minimization” 
then  uses  the  function  "Invert  a  matrix”  to  invert  the  transpose  of  M.  This  inverted  matrix  is 
returned  to  as  the  N  matrix.  "Global  MSE  Minimization”  then  computes  the  optimal  value  for 
each  weight  by  calling  the  function  "Calculate  Weight  Matrix”.  This  function  uses  the  values  in 
the  y,  5  and  inverted  M  matrix  to  perform  the  optimal  weight  linking  node  B  in  layer  1  to  node 
D  in  layer  2,  wbo^  as: 


L  p 

^BD  =  (F.12) 

1=1  p=i 

This  calculation  is  made  for  each  weight  linking  a  node  in  layer  1  to  a  node  in  layer  2. 

F.3.3,2  Make  Nodes  at  Data  Points  This  module  sets  the  weights  of  the  nodes  in  the 
first  layer  equal  to  the  features  of  the  input  patterns.  This  module  begins  by  applying  a  data  record 
to  the  network.  "Nodes  at  Data  Points”  then  calls  the  function  "Calculate  Layer  1  Output”  to 
determine  the  output  for  each  node  in  layer  1.  "Make  Nodes  at  Data  Points”  then  compares  the 
output  of  each  node  in  layer  1  to  the  predefined  threshold  Tmax  and  the  class  of  each  node  in  layer 
1  with  the  class  of  the  input  pattern.  If  the  output  for  a  node  is  greater  than  Tmax  and  the  class 
for  that  node  is  the  same  as  that  of  the  input  pattern,  no  new  node  is  added.  If  not,  a  new  node 
is  added  whose  weights  match  the  input  pattern  features.  This  process  continues  for  each  training 
vector. 


F.3.3,3  Center  Weights  at  Class  Averages  This  module  incrementally  adapts  the  weights 
of  the  nodes  in  layer  1  to  the  running  average  of  the  data  records  lying  within  a  threshold  Tmax 
of  the  current  center.  This  module  begins  by  applying  a  training  vector  to  the  network.  "Cen¬ 
ter  Weights  at  Class  Averages”  then  calculates  the  distance  from  all  the  nodes  in  layer  1  to  that 
training  vector  by  the  equation: 


K 

\t=l 


(F.13) 


If  this  distance  for  a  node  less  than  the  Tmax  j  "Center  Weights  at  Class  Averages”  calls  the  function 
"Update  Average”  to  update  the  current  weights  of  that  node  by  the  equation: 


^ti = ^k\ + 


iV  +  1 


(F.14) 
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If  the  distance  between  a  training  vector  and  all  the  nodes  in  layer  1  is  greater  than  Tmaxt  ”  Center 
Weights  as  Class  Averages”  will  create  a  new  node  and  assign  the  weights  of  the  new  node  the 
features  of  the  training  vector.  This  process  will  continue  until  all  records  in  the  training  set  have 
been  processed.  ^'Center  Weights  at  Class  Averages”  will  repeat  this  process  of  cycling  through  the 
training  data  and  updating  and  creating  new  nodes  until  no  new  nodes  are  created.  At  this  point, 
the  entire  feature  space  is  now  covered  by  the  nodes  in  layer  1. 

F.3.3.4  K- Means  Cluster  This  module  adapts  the  weights  of  the  K  nodes  in  layer 
1  to  be  the  centers  of  K  clusters.  K-Means  Cluster  begins  by  the  setting  the  weights  of  the  K 
cluster  nodes  in  layer  1  equal  to  the  features  of  the  first  K  vectors.  ”K-means  cluster”  then  applies 
a  training  vector  to  the  network  and  calls  the  function  ”Find  Nearest  Neighbor”.  This  function 
computes  the  distance  from  each  input  record  to  each  of  the  K  nodes  in  layer  1  by  the  equation 

K 

di  =  -  Wkif  (F.15) 

This  function  returns  the  reference  number  of  the  node  in  layer  1  with  the  closest  distance  to  the 
input  vector.  ”K-Means  Cluster”  then  assigns  the  input  vector  to  the  cluster  of  the  closest  node 
in  layer  1.  This  process  of  inputting  a  training  vector,  finding  the  node  whose  weights  are  the 
closest  to  that  particular  training  vector,  and  assigning  this  training  vector  to  the  cluster  whose 
node  has  the  closest  weights  is  repeated  until  all  training  records  have  been  processed.  For  each 
cluster  node  in  layer  1,  ”K-means  cluster”  then  takes  the  training  vectors  assigned  to  that  node^s 
cluster  and  repeatedly  calls  the  function  ” Update  Average”.  This  function  updates  each  weight  to 
a  new  average  by  the  equation 


^k-Wkl 

N  +  1 


(F.16) 


After  all  training  patterns  assigned  to  the  cluster  of  a  node  have  been  processed,  each  of  the  node^s 
weights  will  contain  the  average  of  the  features  of  the  pattern  vectors  assigned  to  that  cluster.  After 
this  adjustment,  ”K  means  cluster”  tests  the  difference  between  each  of  the  cluster's  old  weights 
and  new  weights  via  the  equation 


5  =!  I 


(F.17) 


If  the  difference  is  not  less  than  some  small  value  e,  for  all  nodes’  weights,  the  clustering  algorithm 
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has  not  converged  and  this  process  of  applying  a  training  vector,  finding  the  closest  cluster  node, 
and  updating  the  weights  of  the  cluster  nodes  to  be  the  average  of  the  input  pattern  vectors  assigned 
to  the  cluster  will  then  be  repeated.  If  the  difference  is  less  than  £,  the  algorithm  has  converged. 

F.S.8,5  Train  Via  Kokonen  This  module  adapts  the  weights  in  the  first  layer  by  using 
the  Kohonen  algorithm  defined  in  section  3.5.  ^^Train  Via  Kohonen”  first  calls  the  function  ^^Get 
Random  Class  Record”  to  return  a  training  vector  from  the  training  set  with  an  equa*  probability 
according  to  a  uniform  probability  distribution.  ”Train  Via  Kohonen”  then  applies  the  data  record 
to  the  network  and  calls  the  function  ” Calculate  Distance  From  Outputs  To  Next  Layer”.  Assuming 
there  are  K  nodes  in  layer  0  and  J  nodes  in  layer  1,  this  function  performs  the  calculation 

K 

=  (F.18) 

k=l 

These  distance  values  are  stored  and  returned  to  ”Train  Via  Kohonen”  which  then  calls  the  function 
” Find  Nearest  Element”  to  return  the  number  of  the  node  which  had  its  weights  ”closest”  to  the 
features  of  the  input  pattern.  The  module  ”Train  Via  Kohonen”  then  calls  the  function  ”Get  Linear 
Training  Eta”  to  return  the  learning  factor  based  on  the  saw  tooth  function  as  shown  in  figure  F.4. 
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T]  =  (t  -  io)  +  Vmax  (F.19) 

‘0  ""  *max 

Here 

f)  =  the  learning  constant. 

=  the  maximum  77  which  occurs  at  the  beginning  of  the  interval. 

Io  =  the  iteration  number  at  the  beginning  of  the  interval. 

=  the  iteration  number  at  the  end  of  the  interval. 

%  =  the  current  iteration  number. 

Once  the  77  is  returned  ”TVain  Via  Kohonen”  then  executes  the  function  ”Get  Kohonen  Neighbor¬ 
hood”  .  This  function  returns  the  neighborhood  number  of  the  nodes  which  are  in  the  neighborhood 
of  the  winning  node.  The  neighborhood  is  determined  by  the  iteration  number  as  follows: 

neighborhood  =  7  for  0  <  iterations  <  5000 

neighborhood  =  5  for  5000  <  iterations  <  10000 

neighborhood  =  3  for  10000  <  iterations  <  15000 

neighborhood  =  1  for  15000  <  iterations  <  20000 

”T>ain  Via  Kohonen”  then  calls  function  ”Find  Kohonen  Boundaries”  to  check  the  kohonen  layer 
boundaries  to  ensure  that  the  length  of  the  neighborhood  does  not  exceeded  the  boundaries.  ”Train 
Via  Kohonen”  will  determine  which  nodes  will  be  updated  by  invoking  the  function  ” Determine 
Neighborhood  Elements”  and  update  each  of  these  node  weights  by  executing  the  function  ”Train 
Kohonen  Weights”.  This  function  updates  each  of  the  required  nodes  by  the  equation 

wf  =  +  t){xi  -  twf )  (F.20) 

”Train  Via  Kohonen”  repeats  this  functional  execution  for  the  selected  number  of  iterations. 

F.S.3.6  MSB  Remaining  Layers  This  module  performs  the  backpropagation  of  the 
error  to  adjust  the  network  parameters  for  the  weights  and  sigmas  for  each  node  in  the  network. 
”MSE  Remaining  Layers”  first  calls  the  function  ”Get  Random  Class  Record”  to  obtain  a  random 
training  pattern  from  one  of  the  classes  according  to  uniform  probability  distribution.  ”MSE  Re¬ 
maining  Layers”  then  invokes  ”Feed  Forward  Network  Output”  to  calculate  the  network  output. 
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After  the  network  output  is  calculated,  ”MSE  Remaining  Layers”  determines  the  difference  be¬ 
tween  the  desired  network  output  and  the  actual  network  output,  or  error,  by  calling  the  function 
’’Calculate  Errors  In  Output”.  This  function  compares  the  output  of  each  node  in  the  output  layer 
to  their  desired  output  for  that  pattern.  This  comparison  is  done  by  the  following  equation: 


I  2/m  ”  dm  1^  ^max  (F.21) 

Here  tmax  5s  the  maximum  threshold,  currently  set  to  .9,  t/m  is  the  output  for  node  m  in  the  output 
layer  and  dm  is  the  desired  output  for  node  m  in  the  output  layer.  If  all  the  output  nodes  meet 
this  criteria  no  weights  are  updated  and  this  process  is  repeated  for  another  random  record.  If 
any  of  the  output  nodes  don’t  meet  this  criteria,  ”MSE  Remaining  Layers”  will  then  update  the 
remaining  network  layers  parameters  via  backpropagation  according  to  the  MSE  algorithm.  If  the 
transfer  function  for  the  last  layer  nodes  are  sigmoidal,  their  weights  and  sigma  will  be  updated  by 
the  equation 


^MN  =  '%/Ar  +  -  j/Ar)j/w(l  -  J/N)(yitf) 


(F.22) 


and 


<^N  =  <^N  +  -  vn)  (f.23) 

If  the  transfer  function  is  the  linear  transfer  function,  the  node  weights  and  threshold  in  the  last 
layer  will  be  updated  via  the  equations 

wfjpf  =  +  ^{dN  -  2/*Y)(2///f )  (F.24) 

”MSE  Remaining  Layers”  will  then  call  the  function  ”MSE  Mid  Layer”  to  update  the  weights  and 
sigmas  for  the  nodes  in  the  next  to  the  last  layer  of  the  network  via  the  equations 

N 

-^LM  +  ^  -  yn)yn(i  “  yn)«'MnyM(l  “  yM)yL  (F.25) 

and 
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(F.26) 


=  ~  “  yn)wMnyM{l  -  ^Af) 

n=l 

After  the  weights  for  the  next  to  last  layer  have  been  updated,  ”MSE  Remaining  Layer”  will  then 
call  function  ”MSE  1st  Layer”,  if  required,  to  update  the  weights  of  the  first  layer.  Currently,  for 
a  three  layer  network,  all  nodes  must  have  the  sigmoidal  transfer  function.  Therefore  ”MSE  1st 
Layer”  will  update  the  node  weights  and  thresholds  via  the  equations 

N  M 

^KL  =  '^KL  +  ^  -  yn)yn(l  “  yn)[X)  ‘^mnym{l  “  ym)wx,myi(l  “  yi,)yif]  (F.27) 

n=l  m=l 


and 


N  M 

o'l  =  -  yn)yn(i  -  yn)[52  ^mr.ym(i  -  ym)wimyi(i  “  y^)]  (F.28) 

After  the  first  layer  parameters  have  been  updated  ”MSE  Remaining  Layers”  will  continue  this 
backpropagation  until  the  required  number  of  iterations  are  achieved. 

F.3.S.  7  CE  Remaining  Layers  This  module  performs  the  backpropagation  of  the  error 
according  to  the  CE  algorithm  to  adjust  the  network  parameters  for  the  weights  and  sigmas  for  each 
node  in  the  network.  ”CE  Remaining  Layers”  first  calls  the  function  ”Get  Random  Class  Record” 
to  obtain  a  random  training  pattern  from  one  of  the  classes  according  to  a  uniform  probability 
distribution.  ”CE  Remaining  Layers”  then  invokes  ”Feed  Forward  Network  Output”  to  calculate 
the  network  output.  After  the  network  output  is  calculated,  ”CE  Remaining  Layers”  determines 
the  difference  between  the  desired  network  output  and  the  actual  network  output,  or  error,  by 
calling  the  function  ” Calculate  Errors  In  Output”.  This  function  compares  the  output  of  each 
node  in  the  output  layer  to  their  desired  output  for  that  pattern.  This  comparison  is  done  by  the 
following  equation. 


I  Vm  dm  1^  imax  (F,29) 

Here  imax  is  the  maximum  threshold,  currently  set  to  .9,  is  the  output  for  node  m  in  the  output 
layer  and  dm  is  the  desired  output  for  node  m  in  the  output  layer.  If  all  the  output  nodes  meet  this 
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criteria  no  weights  are  updated  and  this  process  is  repeated  for  another  random  record.  If  any  of 
the  output  nodes  don’t  meet  this  criteria,  ”CE  Remaining  Layers”  will  then  update  the  remaining 
network  layers  parameters  via  backpropagation  according  to  the  CE  algorithm.  ”CE  Last  Layer” 
is  called  first  to  update  the  nodes  in  the  last  layer  by  the  equation 

^AfAT  =  “  2/iv)(yAf)  (F.30) 

and 

^iV  =  ^AT  +  '2  (F-31) 

The  factor  of  1/2.3  is  due  to  the  multiplicative  factor  l/ln(10)  from  the  derivative  of  the  CFE 
objective  function.  ”CE  Remaining  Layers”  will  then  call  the  function  ”CE  Mid  Layer”  to  update 
the  weights  and  sigmas  for  the  nodes  in  the  next  to  the  last  layer  of  the  network.  ”CE  Mid  Layer” 
will  update  the  weights  via  the  equation 

AT 

=  ^LM  +  2^  -  yM)yL  (F.32) 

n=l 

and 

AT 

+  2  37V  2/a/)  (F.33) 

n=l 

After  the  weights  for  the  next  to  last  layer  have  been  updated,  ”CE  Remaining  Layers”  will  then 
call  function  ”CE  Last  Layer”,  if  required,  to  update  the  weights  of  the  first  layer.  This  function 
updates  the  via  the  equations 

N  M 

^KL  =  '^KL  +  2^  “  yn)[X)  ^mnym(l  “  ym)wLmyL{l  “  yL)yK]  (F.34) 

n=l  m=l 

and 

N  M 

^L=<^1  +  YzN  “  2/n)[X]  W'mnymCl  “  ym)wLmyLil  “  Vl)]  (F.35) 

n=l  m=l 
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After  the  first  layer  parameters  have  been  updated  Remaining  Layers”  will  continue  this 
backpropagation  until  the  required  number  of  iterations  are  achieved, 

F,3,3»8  CFM  Remaining  Layers  This  module  performs  backpropagation  according  to 
the  CFM  algorithm  to  adjust  the  network  parameters  for  the  weights  and  sigmas  for  each  node 
in  the  network.  ”CFM  Remaining  Layers”  first  calls  the  function  ”Get  Random  Class  Record” 
to  obtain  a  random  training  pattern  from  one  of  the  classes  according  to  a  uniform  probability 
distribution.  ”CFM  Remaining  Layers”  then  invokes  ”Feed  Forward  Network  Output”  to  calculate 
the  network  output.  After  the  network  output  is  calculated,  ”CFM  Remaining  Layers”  determines 
the  node  in  the  output  layer  with  the  largest  output  by  calling  the  function  ”  Determine  Network 
Class”.  This  function  assigns  the  network’s  classification  as  the  largest  value  of  the  nodes  in  the 
output  layer  by  the  equation 

class  -  (F.36) 

where  imax  =  the  output-threshold.  If  the  network’s  class  is  not  the  class  of  the  training  vector  or  if 
the  difference  between  the  correct  node’s  output  and  the  next  highest  node’s  output  is  less  than  the 
user  predefined  difference,  the  network  nodes  are  updated  according  to  the  CFM  algorithm.  ”CFM 
Remaining  Layers”  will  then  update  the  remaining  network  layers  parameters  via  backpropagation 
according  to  the  CFM  algorithm.  ”CFM  Last  Layer”  is  called  first  to  update  incorrect  node  in  the 
last  layer  by  the  equatio: 

'^MN  =  '^MN  -  -  ZN)yN0-  -  yw)(j/A/)  (F.37) 

and 

O’iv  =  “  ^N)yN0-  -  Vn)  (F.38) 

while  the  correct  node’s  weights  and  threshold  are  updated  according  to  the  equation 

^tiC  =  1C “  ^n)!/c(l  -  yc)iyM)  (F.39) 

n=l 

and 
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(F.40) 


cri=wc  +  £  Znil  -  zn)yc{l  -  Vc) 

n=l 

”CFM  Remaining  Layers”  will  then  call  the  function  ”CFM  Mid  Layer”  to  update  the  weights  and 
sigmas  for  the  nodes  in  the  next  to  the  last  layer  of  the  network.  ”CFM  Mid  Layer”  will  update 
the  weights  and  threshold  via  the  equations 


^LM  =  + 


N-1 


N 

X)  -  ^n)[yc(i  -  yc)wMc  “  yn(i  -  yn)wMn]yM(l  “  yM)yL 

n=l 


(F.41) 


and 


=  +  “  •2^n)[yc(i  -  yc)«'Afc  “  yn(i  “  yn)«'Mn)yM(i  “  yw)  (F.42) 


N 


n=l 


After  the  weights  for  the  next  to  last  layer  have  been  updated,  ”CFM  Remaining  Layers”  will  then 
call  function  ”CFM  First  Layer” ,  if  required,  to  update  the  weights  of  the  first  layer.  This  function 
updates  the  weights  and  thresholds  via  the  equations 


and 


M 


n=l  m=l 

M 

““  2/n(l  2/«)  ^  ]  ^mn]ym(l  **  2/m)2/L(l  “  VL^VK 


m=l 


(F.43) 


N 


M 


=  H  +  -^^Y^^n{l-zn)[ycO--yc)'Yj'"”'C 

n=l  m=l 

M 

“  yn(i  “  Vn)  )  j  ^mn]2/m(^  “  J/m)2/L(l  ““  2/l) 


mzzl 


(F.44) 


After  the  first  layer  parameters  have  been  updated  ”GFM  Remaining  Layers”  will  continue  this 
backpropagation  until  the  required  number  of  iterations  are  achieved. 
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F.3,8.9  Scale  Sigmas  by  Class  Interference  This  module  scales  the  sigmas,  for  nodes 
in  layer  1  containing  radial  basis  functions,  if  required,  by  a  constant.  This  module  begins  by 
applying  the  training  vectors  to  the  network  and  calling  the  function  "Calculate  Layer  1  Output” 
to  calculate  the  outputs  from  the  nodes  in  layer  1.  "Scale  Sigmas  by  Class  Interference"  then 
compares  the  output  of  each  node  in  layer  1  with  the  threshold  Tmoa?*  If  the  output  for  a  node  is 
greater  than  Tmajp  and  that  node  is  not  assigned  the  same  class  as  that  of  the  input  pattern,  that 
nodes  sigma  is  scaled  by  the  equation: 

—  Constant)  (F.45) 

"Scale  Sigmas  by  Class  Interference"  will  continue  to  reduce  the  a  for  that  node  until  the  output 
is  less  then  Tmar*  This  process  is  repeated  for  all  the  training  vectors  in  the  training  set. 

F.S.SJO  Set  Sigma  at  P  Neighbor  Avg  This  module  sets  the  sigma  for  each  node  in 
the  first  layer  equal  to  the  root  mean  square  of  the  distance  between  that  node  and  its  P  nearest 
neighbors.  This  module  begins  by  calling  the  function  "Find  Distance  Between  Nodes".  This 
function  calculates  the  distance  between  the  nodes  in  layer  1  by  the  equation: 


K 

\  -  Wki)^ 

'jib=l 


(F.46) 


After  the  distances  between  all  the  nodes  have  been  calculated,  "Set  Sigma  at  P  Neighbor  Average" 
will  then  call  the  function  "Sort  2  Dimensional  Array"  to  find  the  P  shortest  distances  for  each 
node.  "Set  Sigma  at  P  Neighbor  Avg"  will  then  calculate  the  sigma  for  each  node  by  the  equation: 


<Tj  =: 


(F.47) 


F.S.3.JJ  Set  Sigmas  to  Constant  This  module  sets  the  sigma  for  each  node  in  layer  1 
to  a  predefined  constant  C  by  the  following  equation: 


or/  =  Constant  (F.48) 

F,3.4  NETINPUT  This  module  contains  the  functions  necessary  to  load  the  training  and 
test  data  files.  This  data  may  be  loaded  from  a  separate  training  and  test  file  or  a  single  file.  This 
method  of  loading  the  data  is  controlled  by  the  function  "load  input  patterns". 
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Load  Input  Patterns  This  function  allows  the  user  to  control  how  the  test  and 
training  data  is  to  be  loaded.  Currently,  the  function  "Load  Separate  Files"  allows  the  user  to 
load  the  training  and  test  data  from  separate  files  either  in  sequence  or  randomly.  The  function 
"Load  From  Single  File”  allows  the  user  to  load  the  training  and  test  data  from  a  single  file  either 
in  sequence  of  randomly.  The  function  "Load  By  Classes"  allows  the  user  to  control  the  number  of 
training  vectors  assigned  to  ea±  class.  This  is  important  if  the  number  of  training  vectors  must 
reflect  the  a  priori  probability  of  the  input  data.  These  data  vectors  will  be  randomly  selected  from 
a  single  file  of  input  data. 

F.S,4»^  Get  Data  This  function  loads  the  input  data  into  the  training  and  test  data 
structures.  The  training  and  test  data  structures  contain  an  array  of  the  features  of  the  input 
patterns,  the  class  of  the  pattern,  and  the  sequence  number  in  the  input  file. 

F.3.4»S  Normalize  Data  This  function  allows  the  user  to  normalize  the  input  data  via 
the  equation: 


Xi  = 


(F.49) 


F.S.4-4  Get  Weights,  Sigmas,  Classes,  and  Outputs  These  functions  load  initial  weights, 
sigmas,  classes  and  outputs  from  an  input  file.  This  is  important  for  applications  in  which  the  initial 
conditions  of  the  network  are  known  a  priori. 


F.S.5  NETINIT  This  module  contains  the  functions  which  allocate  memory  for  the  nodes 
and  data  records,  correct  node  weights  and  connections,  and  initialize  the  node  weights,  sigmas, 
transfer  functions  and  network  connections. 


F.S,5J  Initialize  Node  Weights  This  function  initializes  the  node  weights  between 
each  connected  node  in  the  network  to  a  value  between  0  and  1.  For  nodes  not  connected,  the 
weights  are  set  to  0. 

F.S.5.2  Initialize  Node  Connections  This  function  connects  the  nodes  in  the  network 
as  defined  by  the  network  type.  Currently,  the  only  network  type  allowed  is  the  feedforward 
network.  For  this  network,  the  nodes  in  layer  0  will  receive  inputs  from  no  other  nodes  and  there 
connections  to  all  other  nodes  will  be  assigned  0.  The  nodes  in  layer  1  will  only  be  connected  to 
and  receive  inputs  from  the  nodes  in  layer  0.  The  nodes  in  layer  2  will  only  be  connected  to  and 


receive  inputs  from  the  nodes  in  layer  1.  The  nodes  in  layer  3  will  only  be  connected  to  and  receive 
inputs  from  the  nodes  in  layer  2. 

F.S,5.3  Iniiialize  Node  Sigmas  This  function  initializes  the  sigma  between  each  node 
in  the  network  to  a  value  between  0  and  1; 

F.3,5.4  Initialize  Node  Outputs  This  function  initializes  the  outputs  for  each  node  in 
the  network  to  a  value  of  0. 

F.3.5,5  Initialize  Node  Transfer  Functions  This  function  initializes  the  transfer  func¬ 
tions,  as  defined  by  the  user,  for  each  node  in  the  network. 

F.3.5.6  Create  Node  This  function  allocates  enough  memory  to  hold  the  ”Node  Data” 
data  structures  for  each  node  in  the  network. 

F. 3,5, 7  Create  Data  Record  This  function  allocates  enough  memory  to  hold  the  ”  Data” 
data  structures  for  each  training  and  test  input  data  vector. 

F,3,5,8  Disconnect  Node  This  function  will  disconnect  any  nodes  in  the  network  from 
all  other  nodes  if  the  node  is  not  being  used.  This  is  important  for  applications  in  which  the  number 
of  nodes  in  the  network  is  not  known  apriori  and  pruned  as  a  result  of  network  learning. 

F,3,5,9  Correct  Node  Weights  This  function  limits  the  range  of  a  node’s  weights  to 
a  value  between  -100  and  100.  This  is  important  to  prevent  backpropagation  training  algorithms 
from  saturating  the  transfer  functions  during  training. 

F.3,6  NETSHOW  This  module  contains  the  output  functions  necessary  to  display  and  file 
the  performance  and  parameters  of  the  network. 

F,3,6,l  Print/File  Network  Output  These  functions  show  the  current  output  for  the 
final  layer  of  nodes  in  the  network. 

F,3.6,2  Print/ File  Data  Parameters  These  functions  show  the  name  of  the  training 
and  test  data  files,  the  number  of  data  vectors  in  each  file,  and  the  dimension  of  the  data  vectors. 
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F.3.6.3  Prini/FUe  Randomization  Rule  These  functions  show  the  randomization  rule 
selected  to  load  the  input  data.  If  the  randomization  rule  is  to  "Load  By  Class",  this  function  will 
also  display  the  number  of  training  patterns  selected  from  each  class. 

F,3,6.4  Prini/FUe  Seeds  These  functions  show  the  initial  weight,  sigma,  data,  and 
record  seeds  used  to  set  the  randomization  rules  for  the  network  parameters.  The  weight  and 
sigma  seeds  control  the  initialization  of  the  network  weights  and  thresholds.  The  data  seed  controls 
the  randomization  of  the  input  data  vectors  while  the  record  seed  controls  the  presentation  of  the 
data  vectors  for  the  backpropagation  algorithms. 

F,3,6,5  Prini/FUe  Net  Topology  These  functions  show  the  current  topology  of  the 
network,  including  the  type  of  network,  the  number  of  layers  in  the  network,  and  the  number  of 
nodes  in  each  layer.  This  is  important  in  the  monitoring  of  the  network  topology  for  networks  in 
which  the  number  of  nodes  is  adapted  according  to  the  data  parameters. 

F,3,6,6  Prini/FUe  Transfer  Functions  These  functions  show  the  transfer  functions  for 
each  layer  of  nodes  in  the  network. 

F.3,6.7  Prini/FUe  Node  ai  Data  Points  Data  These  functions  show  the  number  of 
nodes,  output  threshold,  and  the  initial  sigmas  for  the  nodes  in  layer  1  when  trained  using  this 
training  rule. 


F.3,6»8  Print/File  Center  at  Class-Cluster  Averages  Data  These  functions  show  the 
initial  parameters  of  the  maximum  number  of  initial  nodes,  the  average  threshold  and  sigma  thresh¬ 
old  used  to  train  the  nodes  in  layer  1  using  this  training  rule. 

F,3,6.9  Prini/FUe  K  Means  Data  These  functions  show  the  number  of  clusters  used 
to  establish  the  node  1  weights  when  the  K-Means  algorithm  is  used  to  train  the  weights  for  the 
layer  1  nodes. 

F, 3,6 JO  Prini/FUe  Kohenen  Data  These  functions  show  the  parameters  for  the  num¬ 
ber  of  nodes  in  x  and  y  direction,  the  number  of  training  iterations,  the  neighborhoods,  and  scaling 
factors  used  to  train  a  layer  of  nodes  using  the  Kohonen  algorithm. 

F.3,6J1  Prini/FUe  MSB  Data  These  functions  show  the  parameters  for  the  number 
of  iterations,  the  learning  factor,  and  the  momentum  factor  used  to  train  a  layer  of  nodes  using  the 
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backpropagation  algorithm  according  to  the  MSE  objective  function. 


F.S,6,12  Print/ File  CFM  Data  These  functions  show. the  parameters  for  the  number 
of  iterations,  the  learning  factor,  the  momentum  factor,  the  amplification  factor,  the  offset  factor 
and  the  lateral  shift  factor  used  to  train  a  layer  of  nodes  using  the  backpropagation  algorithm 
according  to  the  CFM  objective  function. 

F.S.6JS  Prini/File  CE  Data  These  functions  show  the  parameters  for  the  number  of 
iterations,  the  learning  factor,  and  the  momentum  factor  used  to  train  a  layer  of  nodes  using  the 
backpropagation  algorithm  according  to  the  CFM  objective  function. 

F. 3.6.14  Print/File  Sigma  Data  These  functions  show  the  training  rules  and  param¬ 
eters  of  the  sigma  threshold,  sigma  constant,  and  number  of  nearest  neighbors  used  to  train  the 
spreads  of  thresholds  of  nodes  having  the  gaussian  transfer  function. 

F.3.7  NETOUT  This  module  contains  the  functions  necessary  to  compute  the  outputs  for 
each  node  in  the  network,  the  outputs  for  each  layer  of  a  feedforward  network,  and  the  output  for 
the  entire  network  due  to  a  given  input  pattern. 

F.3.7. 1  Calculate  Feedforward  Network  Output  This  function  calculates  the  network 
output  due  to  a  given  input  vector  by  calling  the  applicable  output  functions  for  each  layer  of  the 
network. 


F.3.7. 2  Calculate  Layer  0  Output  This  function  establishes  the  range  of  nodes  in  layer 
0  and  calls  ”  Calculate  Node  Output”  to  calculate  the  output  for  each  node  in  layer  0.  This  output 
will  depend  on  the  transfer  function  assigned  to  each  node  in  layer  0. 

F.3. 7.3  Calculate  Layer  1  Output  This  function  establishes  the  range  of  nodes  in  layer 

1  and  calls  ”  Calculate  Node  Output”  to  calculate  the  output  for  each  node*  in  layer  1.  This  output 
will  depend  on  the  transfer  function  assigned  to  each  node  in  layer  1. 

F.3. 7.4  Calculate  Layer  2  Output  This  function  establishes  the  range  of  nodes  in  layer 

2  and  calls  ” Calculate  Node  Output”  to  calculate  '  ^  output  for  each  node  in  layer  2.  This  output 
will  depend  on  the  transfer  function  assigned  to  each  nc  in  layer  2. 
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F.3, 7.5  Calculaie  Layer  S  Output  This  function  establishes  the  range  of  nodes  in  layer 
3  and  calls  ”  Calculate  Node  Output”  to  calculate  the  output  for  each  node  in  layer  3.  This  output 
will  depend  on  the  transfer  function  assigned  to  each  node  in  layer  3. 

F,S.7.6  Calculate  Node  Output  This  module  computes  the  output  for  each  node  in  the 
network  as  determined  by  the  transfer  function  and  the  other  nodes  from  which  the  node  receives 
input.  Currently,  a  node^s  transfer  function  can  either  be  the  sigmoidal  transfer  function  of 


(F.50) 


or  the  gaussian  radial  basis  function  of 


Vout  —  ^ 


(F.51) 


or  the  linear  transfer  function  of 


L 

Vout  =  (F.52) 

/=i 

or  the  identity  transfer  function  in  which 


Vout  —  Vin  (F.53) 

F.S.8  NETAUX  This  module  contains  the  functions  called  by  NETTRAIN  to  set  the  net¬ 
work  parameters.  Below  is  the  list  of  functions  maintained  in  this  module. 

1)  Determine  Y  Matrix 

2)  Determine  S  Matrix 

3)  Determine  M  Matrix 

4)  Calculate  Weight  Matrix 

5)  MSB  Last  Layer 

6)  MSE  Last  Layer  Linear 

7)  MSE  Last  Layer  Sigmoid 

8)  MSE  Mid  Layer 
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9)  MSE  1st  Layer 

10)  Calculate  Errors  in  Output 

11)  Get  Linear  Training  Eta 

12)  Get  Kohonen  Neighborhood 

13)  Calc  Dist  Outputs  to  Nxt  Lyr 

14)  Find  Nearest  Element 

15)  Find  Kohonen  Weights 

16)  Determine  Neighborhood  Elements 

17)  Train  Kohonen  Weights 

18)  Find  Distance  Between  Nodes 

19)  Sort  2  Dim  Array 

20)  CE  Last  Layer 

21)  CE  Mid  Layer 

22)  CE  First  Layer 

23)  Calculate  Zn 

24)  CFM  Last  Layer 

25)  CFM  Mid  Layer 

26)  Find  Second  Highest  Node 

27)  CFM  First  Layer 

28)  Find  Nearest  Neighbor 

For  a  detailed  discussion  of  these  functions,  see  NETTRAIN  or  Appendix  G. 

F,3.9  NETMATH  This  module  contains  the  mathematical  functions  used  by  the  various 
training  algorithms  within  the  module  NETTRAIN. 

F.3.9J  Make  Ideniiiy  Matrix  This  function  returns  a  square  identity  matrix  to  the 
calling  routine.  The  number  of  rows  and  columns  in  the  matrix  and  the  address  of  the  matrix  must 
be  passed  to  this  function  for  proper  execution. 
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F.3,9.2  Determine  Matrix  Transpose  This  function  returns  the  transpose  of  a  matrix. 
The  calling  function  must  provide  the  addresses  of  the  matrix  to  be  transposed  and  the  matrix 
transpose,  along  with  the  number  of  rows  and  columns  of  the  matrix. 

F,S.9.S  Invert  A  Matrix  This  function  returns  the  inverse  of  a  square  matrix.  The 
calling  function  must  provide  the  addresses  of  the  matrix  to  be  inverted  and  the  final  inverted 
matrix  along  with  the  number  of  rows  in  the  matrix.  The  matrix  inversion  is  completed  via 
gaussian  elimination. 

F.3.9,4  Update  Average  This  function  computes  the  running  average  of  a  series  of 
data  points.  The  calling  function  must  provide  the  current  average,  the  next  data  point  to  be 
incorporated  into  the  average  and  the  number  of  data  points  within  the  average.  The  update 
equation  is  as  follows: 


+  ^(®‘'  -  «r )  (F-54) 

F,3,9.5  Update  Sigma  This  function  computes  the  running  standard  deviation  of  a 
series  of  data  points.  The  calling  function  must  provide  the  current  average,  the  next  data  point 
to  be  incorporated  into  the  standard  deviation,  the  current  standard  deviation  and  the  number  of 
data  points  in  the  calculation.  The  update  equation  is  as  follows: 

(r+  =  +  ~)ixi  -  w)2  -  al)  (F.55) 

F,3.9,6  Calculate  Percentage  This  function  computes  the  percentage  a  ratio  of  num¬ 
bers.  The  calling  function  must  provide  the  numerator  and  the  denominator  of  the  numbers. 

F, 3.9.1  Get  Random  Class  Record  This  function  randomly  selects,  according  to  a 
uniform  probability  distribution,  one  of  the  possible  classes  and  randomly  returns  a  data  vector 
from  the  selected  class. 

F.3.9.8  GhJ  Random  Record  This  function  randomly  selects,  according  to  a  uniform 
probability  distribution,  a  data  vector  from  the  set  of  training  data. 
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Appendix  G.  Software  Code 


G.l  NETMENUE 


/*****0************M*******.*************».***4******0*****0****/ 

/•  Hodul*  laae:  lETHEIUE  luabar:!  */ 
/*  Description:  This  module  is  the  overall  controlling  module  of  e/ 
/e  of  the  software  a  It  provides  the  user  interface  */ 
/*  to  the  softvare  via  a  SUI  workstation  terminal  */ 
/*  and  keyboard.  This  module  calls  the  appropriate  */ 
/w  modules  to  execute  user  desisions.  */ 


/*  Modules  Called:  IETERR0R»  lETTRAII,  lETIlPUT,  lETIlIT,  lETSBOV  «/ 
/e  Functions  Contained:  lone  e/ 

/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 

•include  "netvrble.h" 

•include  "netfnctn.h” 

mainO 

{ 

FILE  efptr,  etrain^ptr,  etest.ptr,  euser.ptr; 

FILE  ♦MSE.ptr,  ♦CB.ptr,  eCFM^ptr; 

struct  data  e training.dat a [TRAII^SET] ; 

struct  data  etest.dataCTEST.SET] ; 

struct  lode.data  e|ode.record[TOTAL.VODES] ; 

float  output .threshold  >  1; 

float  average.threshold  «  2; 

float  sigma.threshold  -  4; 

float  sigma.f actor  »  .5; 

float  interference.threshold  »  .8; 

float  MSE.error.delta  »  .2; 

float  class.threshold  ^0; 

float  per.cent.correct  »  0; 

float  sigma.constant  ~  2; 

float  MSE.eta  *  .3; 

float  CFM. alpha  =  1; 

float  CFM.beta  =  4; 

float  CFM.eta  »  3; 

float  CFM.zeta  «  0; 

float  CFM.momentum  ^  .1; 

float  CFM.delta  ^  .4; 

float  CE.epsilon  »  .1; 

float  CE.it orations  »  20000; 

float  CE.momentum  =  *1; 

float  CE.eta  =  2.76; 

float  MSE.momentua  -  .1; 

static  int  neighborhoods [6]  =  {7,5,3,!}; 

static  int  train. width [6]  =  {0,  5000,  10000,  15000,  20000}; 

static  float  train.scale [6]  =  {.1,  .05,  .025,  .0125}; 

int  misclassified[200] ; 

int  train.error  =  0; 

int  tost.error  =  0; 

int  total.error  =  0; 

int  correct.class  =  0; 

int  network.class  -  0; 

int  width.no  =  5; 

int  nodes.x  =  10; 

int  nodes.y  =  0; 

int  p.neighbors  =  0; 

int  train.set  =  TRAII.SET; 

int  test. set  =  TEST.SET; 

int  classes  ~  CLASSES; 
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itit  dinensioxi  ^  DINEISIOI; 
int  node; 

int  record,  row,  x,  y,  layer,  8tarting.node.in.layerC4] ; 
int  number^of .layers ; 

int  nodes.in.layerC4] ,  training.rule [4] ,  transf er.iunction[4] ; 

int  network.type  «  0; 

int  total.nodes  »  0; 

int  preset  «  0; 

int  sigma.rule  -  0; 

int  int.buffer  »  0; 

int  total.iterations  -  0; 

int  current.node  =  0; 

int  HSE.iterations  *  0; 

int  kohonen.iterations  »  0; 

int  nodes.l.uax  -  0; 

int  node8.2.iaax  s  O; 

int  nodes.3.inax  »  0; 

int  error  “  0; 

int  CFH.sttccesses  =  100; 

int  CFM. iterations  =  20000; 

int  CE.successes  ==  100; 

int  randomization.rule  «  0; 

int  training.pattems.in.classCCLiSSES] ; 

int  MSE.successes  =  100; 

int  class  =  0; 

int  find.the.distance  -  0; 

int  normalize.the.data  »  0; 

unsigned  data.seed  ^  0; 

unsigned  signa.seed  -  0; 

unsigned  wght.seed  ~  0; 

unsigned  record.seed  =  0; 

/^^*t******i^****t*^ES^  STUFF 

static  char  train.fileC]  =  **class2 . in" ; 
static  char- test .file □  =  "class2.in"; 
static  char  output .filc[]  =  "nodes.test.out"; 
static  char  selection_file[]  =  "nodes.test.sel"; 
static  char  MSE.file[]  ^  "MSE.data.out"; 
static  char  CFH.file[]  =  '‘b6ta4data"; 
static  char  CE.fileC]  =  "CE.data.final”; 


norm2dize.the.Qa;a  =  0; 

/♦la  yes  ♦/ 

find^the.distance  »  0; 

/♦  1  =  yes  ♦/ 

dinension  »  60; 

train.set  *=  102; 

test.set  =  100; 

classes  =  2; 

/*  Randomization  Rule 

♦/ 

randomization.rule  =  3; 

/*  1  -  load  sepurate.files 

*/ 

/♦  2  load  from  single -file 

^  t 
^  / 

/*'  3  -  load  by  class 

♦/ 

training.patterns.in.classClj.  =  51; 
trainihg.patterns.in.class  [2]  =  51 ; 
training.patterns.in.(^las8[3]  =  0; 
training.patterns.in.classC4]  =  6;^ 
-tr'aining.patterns^in^clasBES]  =  0; 
training:,patternSiinicla88CC]  =  -0; 

^raining.patterns.in.c3  ass  [73  =  0 
t'raining_pattern8.in.clas£.[8j  *  0; 
training.patterns.in.clasoEO]  =  0; 
train irig.pat t  erns.  in.class [10]  =  0 ; 

wght.seed  =  0;  sigma.seed  =  0;  data.soed  =  1;  record.seed  =  1; 
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netvork.type  =  1;  number.of flayers  =  2; 

nodes.'in.layerCO] -dimension; 
nodes_in.lay «r [1] =60 ; 
nod€8.in.lay er [2] =2 ; 
nodes. in.layer [3] =0 ; 

training.rule [0] =0 ; 

training.rule [!]=! ; 

/«  l-nodes  at  data  points  2-center  class  average  3-  K-means  */ 

/♦  sig-thres,  out-thres  avg-thresh  sigthresh  sig  rule  3or4  ♦/ 

/*  4-kohonen  S-MSK  backprop  6-CFH  backprop  7-CE  backprop  ♦/ 

/♦  nodes.x  MSE  stuff  CFM  stuff  CE  stuff  */ 

.  raining.rule  [2]  =  1; 

/♦I  -  matrix  invert  2  -  MSE  backprop  3-CFM  backprop  4-CE  backprop  ♦/ 
/♦  S  -  Parzen  window  MSE  stuff  CFM  stuff  CE  stuff  ♦/ 

trainlng.rule [3]  =  0; 

/♦I  -  MSE  backprop  2  -CFM  backprop  3-CE  backprp  4-Parzen  window  ♦/ 

/♦  MSB  stuff  CFM  stuff  CE  stuff  ♦/ 

sigma. threshold  =  4;  kohonen. it orations  =  20000; 

output  .threshold  =  1;  nodes.x  =  7; 

average.threshold  =  2; 

MSE.it  orations  =  30000;  CFH.alpha  =  1.0;  CE.epsilon  =  .05; 

hSE.error.delta  =  .1;  CFM.bota  =  4.0;  CE.iterations  =  30000; 

MSEjaomentum  =  .1;  CFM.eta  =  .14;  CE.Eomentum  =  .05; 

MSE.eta  =  .3;  CFM.zeta  =-0;  CE.eta  =  1.75; 

MSE.successes  =  100;  CFM.successes  =  100;  CE.successes  =  15000; 
CFM. iterations  =  150000; 

CFH.momentum  =  .1; 

CFM.delta  =  1.0; 

transfer.functionCO]  =0;  /♦  1-  sigmoidal  ♦/ 

transfcr.functionCl]  =2;  /♦  2  -rbf  ♦/ 

transfer.f unction [2]  =  3;  /♦  3-  linear  ♦/ 

transfer. function [3]  =  0; 


sigma.rule  =1;  /♦  Sigma  rules  1  -  scale  by  constant  ♦/ 

/♦  interference.threshold  =  .8;  ♦/ 

/♦  sigma.f  actor  =  .5;  ♦/ 

interference.threshold  =  .4; 

sigma.f  actor  =  .1;  /♦  2  -  half  nearest  neighbor  ♦/ 

sigma.constant  =  .5; 

p.neighbors  =6;  /♦  3  -  constant  ♦/ 

/♦  sigma.constant  =  2;  ♦/ 

/♦  4  -  p  neighbr  average  ♦/ 

/♦  p.neighbors  =4;  ♦/ 


for  (x  =  0;  X  <  train.set;  x++) 

i 

create.data.record(training.data, 

X, 

terror) ; 

if (error  !=  0) 
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{ 


printf ('’\n*****  out  of  memory  for  training.data 
exitO; 

} 

} 

for  (x  =  0;  X  <  test^set;  x++) 

{ 

creat€.data.record(te8t^data> 

X, 

terror) ; 

ifCerror  !=  0) 

{ 

,printf C'Xne***  out  of  memory  for  test  data 
exitO; 

} 

} 


train.ptr  =  fopen(train«file,"r") ; 
test^ptr  =  fopen(te8t«filo,'*r") ; 

load.input. pat terns  (training.data, 
test^data, 
train.set, 
test.set , 
dimension I 
classes, 

training^pattems.in.class, 

randomization.rule , 

data.seed, 

train.ptr, 

test.ptr) ; 

f close  Ctrain.ptr) ; 

f close(test^ptr) ; 

if  (normalize^the.data  ==  1) 

{ 

normalize.dataCtraining.data, 
train.set, 
dimension) ; 

norm aliz6.dat a(test. data, 
test.set , 
dimension) ; 


if (find«the_distance  ==  1) 

{ 


fptr  =  fopen("nrmtrain.dat'*,"w") ; 

fprintf (fptr,‘'\n  normalized  Distances  for  Training  data"); 
calculate.euclidean.distance.betveen.inputsCtraining.data, 

train.set, 
dimension, 
fptr) ; 


f close (fptr); 

fptr  =  fopenC'nrmtest  .dat”,"w'*) ; 

fprintf (fptr, "\n  normalized  Distances  for  Test  data"); 
calculate.euclidean.distance.between.inputs (test .data, 

test.set, 
dimension, 
fptr) ; 


G-4 


fclosedptr); 

} 

user.ptr  =  fopen(sGlection«file,”w") ; 

nodes.l.max  *  nodes.in.laycrCl] ; 
nodes. 2.tQ&x  »  node8.in.layer[2]  ; 
nodes.3.nax  «  nodes.in.layer[3] ; 

total.nodes  «  nodes. in.layer[0]j 
start ing.node.in.layerCO]  =  0; 

for  (layer  =  1;  layer  <  number.of .layers  +1;  layor++) 

{ 

St  art  ing.node.in.layer  [layer]  =  st  art  ing.node.in.layer  [layer-1] 

+  nodes.in.layer(layer-l] ; 
total.nodes  =  total.nodes  +  node s.in.layer [layer]; 

} 

error  =  0; 

for  (node  »  0;  node  <  total.nodes;  node++) 

{ 

create.nod6  (lode_record> 
node> 
terror) ; 

if  (error  !=  0) 

printf ('*\nout  of  memory"); 
exitO ; 

} 

} 


initialize_node.connections(Iode.record, 

number. of .layers , 

nodes.in.layer, 

st  dLit  ing.node.in.lay  er , 

notyork.type, 

total.nodes) ; 

initialize.node.veights(Iode.r6cord, 
total.nodes, 
wght.seed) ; 

initialize_nod6.sigmas(3ode.record, 
total.nodes, 
sigma^seed) ; 

initialize_node .outputs (Kode.record , 
total.nodes) ; 

initialize.node.transfer.function(Node.record, 

number. of .layers , 
nodes. in.layer, 
starting.node.in.layer, 
transfer .function) ; 


f  ile.data.paramet  ers  (train.f  ile , 
test.file, 
train.set , 
test.set, 
dimension, 
classes. 
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user^ptr) j 


f  il«.randomizatioh.nile  (randomization.riile , 

tzaining^patt  erzis.izi^class , 
classed » 
iiser.ptr) ; 


f  Hayseeds  (vght.saad, 
sigma^sead, 
data.seed, 
racord.8aed> 
usar^ptr) ; 

f  pr  inti  (us  ar.pt  r,*‘\xv8tar  ting  network  topology"); 
f  ila.nat. topology  (natffork.type  > 

number.ol. layers , 
nodes.in.layer , 
user.ptr) ; 

lila_tran8fer.function8(network.type, 
number.of .lay ers , 
starting.node.in.layer, 
lode.racord, 
user.ptr) ; 


for  (layer  =  1 ;  layer  <  nunber.of .layers  +1 ;  layer++) 
if  (layer  ==  1) 

switch  (training.rule [layer]) 

{ 

case  1: 

make.nodes.at.data.points  (training.dat a ^ 

Hode.record, 
train.set, 
node8.in.layer , 
s igma.t hre shold , 
output .threshold , 
start ing_node.in_layer , 
total.node3j 
node8.1.uar) ; 


f  ile.nodes.at.dat  a.points.  inj’o  (layer , 
output . chre  shold , 
sigma.threshold^ 
user.ptr) ; 


break; 
case  2: 

center.weight  s.at.class.averages (training.data , 

Vode.record, 

train.sot, 

nodes.in.layer, 

aver age. threshold, 

cigsa.thrc shold, 

start ing.node.in.layer, 

total.nodes, 

layer, 

nodes.l  jaar) ; 


f ile.center.at.class.avg3.dat a (layer , 
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averag«^thre8hold 
sigma. threshold, 
user.ptr) ; 


break; 
case  3: 

k.meaxic_cluster(traixilxig.data, 
Rode.record, 
train.set , 
nodes.in.layerCl] , 
nodes.in.layer , 
start ing.node.in.layer , 
layer) ; 

file.k.means.data(layer, 

nodes.in.layerCl] , 
user.ptr) ; 

break; 
case  4: 

nodes.y  =  nodes_in.layerCl]/nodes.x; 

train.via -kohonenCtrainizig.data , 
lode.record, 
nodes.in.layer , 
start ing.node.in.layer , 
neighborhoods, 
train.eidth, 
train.scale, 
vidth.no, 
nodes.x, 
nodes.y, 
layer, 
train.set , 
kohonen.iterat ions , 
record.seed) ; 

f ile.kohonen.dataClayer, 
nodes.x, 
nodes.y , 
nser.ptr) ; 


break; 


case  5: 

MSE.ptr  =  fopen(MSE.file,'’v«); 

MSE.remaining_layers(Hode.record, 

training.data, 

test.data, 

t ransfer.f unction, 

nodes.in.layer, 

start ing.node.in.layer, 

number. of .layers , 

layer, 

train.set, 

test.set, 

MSE.eta, 
total.nodes, 
MSE.successes, 
MSE.error.delta , 
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HSE.it er at ions > 
MSE^nomantum, 
classes, 
record.seed, 

MSB.ptr) ; 

f close (MSE.ptr) ; 

f ile.MSE.data(layer , 

HSE.iterations, 

HSE.error.delta, 

HSE^monentna, 

MSE.successes, 

HSE.eta, 
user.ptr) ; 

layer  -  nnmber.of .layers ; 
break; 
case  6: 

CFM.ptr  a  fopenCCFM.file, •’»'*); 

CFH.r  einaining.lay  ers  (lode.record , 
training.dat a, 
test.data, 
nodes.in.layer , 
starting.node.in.layer, 
nuttber.of .layers , 
layer, 
train.set, 
test.set, 

CFM. eta, 
total.nodes, 
CFH.successes, 
CFH^iterations , 

CFN.  alpha, 

CFM.beta, 

CFM.zeta, 

CFH.iaomentum, 
CFH.delta, 
classes , 
record.seed, 

CFM.ptr) ; 

f close (CFM.ptr) ; 

f ile.CFH.dataClayer , 

CFH. alpha, 

CFH.beta, 

CFM.eta, 

CFM.zeta, 

CFH.successes, 

CFM.it erat ions, 

CFM.iaoDentuin, 

CFM.dclta, 
user.ptr) ; 

layer  =  number.of .layers ; 
break; 
case  7: 

CE.ptr  -  fopen(CE.file,*'w") ; 

CE.refflaining.layer8  (lode.record. 
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tr&ining.data, 

test^data, 

nodas.in.layer, 

start  ing.node.iii.layer , 

ntmber.of  .layers , 

layer, 

train.8et, 

test. set, 

CE.eta, 

total.nodes, 

CE.sttccesses, 

CE.epsilon, 

CE.iterations, 

GE.iaomentum, 

classes, 

record.seed, 

CE.ptr) ; 


fclose(CE.ptr); 

f ile.CE.dat aClayer, 
CE.epsilon, 
CE.iterations, 
C£.moBientum, 
CE.eta, 
CE.successes, 
user.ptr) ; 

layer  =  number. of .layers; 
break; 

default : 
break; 


} 

if  (nodcs.in.layerCl]  <  nodes.l.max) 

■I* 

int.buffer  =  nodes.l.mar  -  node s_in.layer [layer] ; 
for  (x  =  0;  X  <  int.buffer;  x++) 

current .node  =  starting.nodejin.layer [layer] 

+  node s.in.layer [layer]  +  x; 
disconnect .node ( Vode.record , 
current^node, 
total.nodes) ; 

} 

} 

if  (Iode.record[startingjiode.in.layer[l]]->transfer.f  unction  ==  2) 

{ 

file.sigma.data (layer, 

sigma.rule, 

int erf erence.threshold , 
sigma.factor, 
sigma.constant, 
p.neighbors, 
user.ptr) ; 

if  (sigma.rule  ==  1) 

scale.sigmas.by.class.interference(training.data, 

lode.record, 

train.set, 

nodes.in.layer. 
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start ing^nods^in^lay er , 

total.nod«8» 

layer, 

interference.threshold, 

sigma.factor); 

> 

else  if  (sigma.rule  ==  2) 

{ 

printf(“\n  error  in  sigma  rule  selection"); 
exitO; 

} 

else  if  (sigma.rnle  ==  3) 

{ 

set^sigmas.to.constantClode.record, 

nodes.in.layer, 

St  art ing«node_in.layer , 
layer, 

sigma.constant) ; 

} 

else  if  (sigma.rnle  ==  4) 

{ 

set.sigma.at.P.neighbor.avgdode.record, 

nodes.in.layer , 

start ing.node.in_layer, 

layer, 

total.nodes, 
p.neighbors) ; 

} 

} 

else  if  (layer  ==  2) 

{ 

switch  (training.rule [layer] ) 
ca$o  1: 

global.HSE.minimizat  ion<  t  r  aining.dat  a , 

Vode.record, 

jrain.set, 

nodes.in.layer, 

St  artivig.node.in.lay  er 

total.rode8, 

layer) ; 


file.matrix.data(layer, 

nser.ptr) ; 

break; 
case  2: 

rr^B^ptr  =  fopen(KSE.file,"»"); 

MSE.reraaining.layers(I'  de.t«cCi“<i» 

trainihg.dat a, 
test.data, 
transfer.f unction , 
nodes.in.layer , 

layer, 

lionber.of .layer/x , 
layer, 
train.set, 
test.set, 

MSE.eta , 
total;,/*oded, 


HSE.sttccesseS) 
HSE.«rr or.delta , 
HSE. iterations, 
KSE^Bomontim, 
classes, 
record^seed, 
MSE.ptr) ; 

f close (HSE.ptr) ; 

file^MSE.datadayer, 

MSE.it orations, 
MSE.error.delta, 

HSE.nomentum, 

NSE.sttccesses, 

MSE.eta, 
nser.ptr) ; 

layer  ®  number.of. layers; 

break; 

case  3: 

CFM. ptr  »  fopen(CFM.file,»»'0; 

CFN. remaining.lay6rs(Iode.record, 

training.dat a, 

test.data, 

nodes.in.layer, 

St  art ing.node.in.lay er , 

nvuaber.  of  .layers, 

layer, 

train.set, 

test.set , 

CFM.eta, 

total.nodes, 

CFM.successes, 

CFM. iterations, 

CFN. alpha, 

CFM.beta, 

CFM.zeta, 

CFM.uomentujtt, 

CFM.delta, 

classes, 

record.seed, 

CFM.ptr) ; 


fclose(CFM.ptx) ; 

f ile.CFM.dat  a(lay er , 

CFM. alpha, 

CFM.beta, 

CFM.eta, 

CFM.zeta, 

CFM.successes, 

CFM.iterations, 

CFM.momentU2a, 

CFM.delta, 

nser.ptr) ; 

Hayer  =  number.of .layers ; 
break; 
case  4: 

CE.ptr  -  fopen(CE.filo,''w»); 
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CE«r6m&ining.l&jQrs  (lode.record , 
training^data, 
test^data, 
nodes.in.layer, 

St  art ing.nod  e^in^lay  er , 

nunbar.  of  .layers  > 

layer » 

train.set, 

test.set, 

CE.eta» 

total.nodes, 

CE.sttccesses, 

CE.epsilon, 

GE. iter at ions > 
CE.noaentuta^ 
classes ) 
record.seed^ 

CE.ptr) ; 


f close (CE.ptr) i 

f  ile.CB.datadayer , 

CE.epsilon, 

CE. iterations, 
CE.nomentuitt, 
CE.eta, 
CE.successes, 
user.ptr) ; 

layer  =  nnaber.of .layers ; 
break; 


case  5: 

PSI.last.lay er (Bodo.record , 

nodes. in.layer, 
start ing.node.in.lay er, 
layer) ; 

fiXe.parzen.vindow.data  (Sode.record, 

nodes.in.layer, 
starting.node.in.layer , 
layer, 
user.ptr) ; 

break; 


default : 

break; 

} 

} 

else  if (layer  ==  3) 

{ 

switch  (training.nile [layer] ) 

{ 

case  1: 

MSE.ptr  =  fopen(MSE.file,»w"); 

HSE.remaining.layers(Hode.record, 

training.dat a, 
test.data, 
t ransfer.f unction, 
nodes.in.layer, 
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start  ixig^node.in.layer , 

nuAber.of flayers ,  . 

layar, 

train^set, 

test.sat, 

MSE^ata, 

total.nodas, 

HSE.succassas^ 

MSE.error.dalta, 

HSE. it a rat ions, 
MSE.nomantua, 
classas, 
racord^saad, 

MSE.ptr) ; 


f close (MSE^ptr) ; 

file.HSE.datadayar, 

MSE.iterations , 

MSE.arror.dalta, 

HSE^moaentun, 

HSE.succassas, 

!!SE^ata, 
usar^ptr) ; 

layer  «  nuabar.of flayers; 

break; 

case  2: 

CFM.ptr  =  fopenCCFM.fila,  •’»"); 

CFIf.rauaining.layars(Ioda.racord, 
tralning.dat  a, 
tost .data, 
nodes.in.layar, 
8tar_ting.noda.in.layer, 
nufflber.of .layers , 
layer, 
train.set , 
test.set, 

CFM.eta, 

total.nodes, 

CFH.successes, 

CFK. iterations, 

CFM. alpha, 

CFM.beta, 

CFH.zeta, 

CFM.mofflentum, 

CFM.delta, 

classes, 

record.seed, 

CFM.ptr) ; 


fclose(CFM.ptr); 

file.CFH.dat a (layer, 

CFM.alpha, 

CFM.beta, 

CFM.eta, 

CFM.zeta, 

CFM.snccesses, 

CFM. iterations , 

CFM.&omentma, 

CFM.delta, 

nser.ptr) ; 
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lay or  =  numbor.of^layorb; 
break; 
case  3: 

CE.ptr  «  fopen(CE«lilo,'‘w"); 

CE.remainlng.layers  (lode.record, 
tralning.data, 
test.dat a, 
nodes.in.layer, 

St art ing^nodo.in.lay or , 

nujnber.of  .layers , 

layer, 

train.set , 

test. set , 

CE.eta, 

total.nodes, 

CE.successes, 

CE.epsilon, 

CE.iterations, 

CE.inoinentvia, 

classes, 

record.seed, 

CE.ptr) ; 


fclose(CE.ptr); 

file.CE.dataClayer, 

CE.epsilon, 

CE.iterations, 

CE.momenttua, 

CE.eta, 

CE.snccesses, 
user.ptr) ; 

layer  =  number. of .layers; 
break; 
case  4: 

PMI.last. layer (Mode. record , 

nodes.in.layer, 
starting.node.in.layer, 
layer) ; 

filc.p  'rzen. to li\dotf.dat a  (Mode.record, 

nodes.in.laycT, 

art ing.node.in.lay er , 
layer, 
usor.ptr) ; 

break; 

default: 

break; 

> 

} 

X 

f print"! (us er.pt r,"\n  Final  topology"); 
file^not.t-opologyCnetvork.typn, 

nuffibeivof  .layers , 
nodes.in-la^ or, 
si-'cr.ptr'> ; 
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fptr  =  foperi(output_filo,*'w’0 ; 
f ilo.net vork.paranet ers (lode^record , 
netBork.type, 
nujaber^of.laycrs , 
nodes.in.layor, 
training.iula, 
transf er.f unction , 
sigma.rule* 
total.nodes, 
f ptr) ; 


for  (node  =  0;  node  <  total.nodes;  node++) 
f ilo.node^data(Hode.record , 
node, 

total.nodes, 
fptr) ; 

for  (x  =  0;  jc  <  classes+l;  x++) 
training_patterns.in.class[x]  =  0; 

train^error  =  0; 

for  (x  a  0;  X  <  train^set;  x++) 

{ 

class  =  training_dataCx3">class; 
training«pattems«in_class  [class]  +=1 ; 

t.est.thG«notHork(training«data, 

Xode.record, 
nodos_in_laycr, 
start ing^node.in.layer, 
n>uaber«of  flayers , 

total.nodes, 
class.threshold , 
misclassified, 

Jttrain.error) ; 

file.last.layer.outputCtraining.data, 

Bode.record, 

x, 

nodes.in.layer, 
start ing.node.ln.layer, 
niimber.of  .layers , 
fptr) ; 


} 

co^rect.class  =  train.set  -  train.error; 
calculate.percentageC (float )correct. class , 
(float ) train.sot , 
ftper.cent.correct) ; 

fprintf (user.pt r,"\ntraining  data") ; 
f ilo.error.dataCtrain.orror , 

per.ccnt.correct , 
misclassified, 
user.ptr) ; 


f ile.class.count (training.patterns.in. class , 
classes, 
user.ptr) ; 
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for  (x  =  0;  X  <  classes+1;  x++) 
training.pattems.in.classCx]  =  0; 

test^error  =  0; 

for  (x  =  0;  X  <  test.set;  x++) 

{ 

class  =  test.data[x3->class; 
training_pattenis.in_classCclas8]  +=  1; 

test.the.netvorkCtest.datai 

lode.record) 

nodes.in.layer> 

St  art ing.node.in.layer , 
nvusber.of  .layers , 

X, 

total.nodes, 

class.threshold, 

misclassified, 

Jttest.error) ; 

file.last.laycr.output(test_data, 

Hode.record, 

X, 

nodes. in.layer, 
start ing.node.in.layer, 

*  nnmber.of. layers, 
fptr) ; 

} 

correct.class  =  test.set  -  test.error; 
calculate.percentageC (float) correct. class , 

(float)test.set, 

Jtjer.cent.correct) ; 

fprintf (user.pt r,*’\ntest  data") ; 
f ile.error.data(test.error, 

per.cent.correct , 
misclassified, 
user.ptr) ; 

file.class.count  (training.pattems.in.class, 
classes, 
user.ptr) ; 

total.error  =  train.error  +  test.error; 
correct.class  =  train.set  +  test.set  -  total.error; 

calculate.percentage ( (float) correct.class, 

(float) (test.set+train.set) , 

Jkper.cent.correct) ; 

fprintf  (fptr," \ntotal  per  cent  correct  =  */.f ", per.cent.correct)  ; 
fprintf  (user.ptr,  "\n  total  per  cent  correct  =  y, f",per.cent. correct)  ; 

f close (fptr) ; 

for  (node  =  0;  node  <  total.nodes;  node++) 
f ree (♦lode.recordCnode] ) ; 
f close (user.ptr) ; 

} 
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G.2  NETERROR 


/♦  Module  Vame:  SETERROR.C  Humber:  2.0  ♦/ 

/♦  Description:  This  module  contains  the  functions  which  provide  ♦/ 

/♦  error  accounting  of  the  network  performance.  ♦/ 

/♦  ♦/ 

/♦  Modules  Called:  5ET0UT  ♦/ 

/♦  Functions  Contained:  2.1  test_the_network  ♦/ 

/♦  2.2  detemine.class^as.lairgest  ♦/ 

/*  2.3  update^errors  ♦/ 

/♦  Date:  11  Hov  90  Revision:  1.0  ♦/ 

tinclude  ’’netvrble.h" 
tinclude  “netfnctn.h” 

/*  Function  Name:  test_the.network  Humber:  2.1  */ 

i*  Description:  This  function  calculates  the  errors  from  a  */ 

/*  feed  forward  network  due  to  an  input  record  */ 

/♦  ♦/ 

/♦  Functions -Called:  7.1  calculate.feed.forware.network. output  ♦/ 

/♦  2.2  detezmine.class^as.largest  ♦/ 

/♦  2.3  update.errors  ♦/ 

/♦  ♦/ 

/♦Variables  Passed  In:  training  or  tost.data  -  Structure  array  ♦/ 

/♦  Hode^record  -  Structure  array  ♦/ 

/♦  nodes. in^layer  -  Integer  array  ♦/ 

/♦  starting.node.in.layer  -  Integer  ♦/ 

/♦  number.of.layers  ~  Integer  ♦/ 

/♦  total.nodes  *•  Integer  ♦/ 

/♦  class.threshold  -  Float  ♦/ 

/♦  misclassified  -  Integer  array  ♦/ 

/*  ♦error  -  Integer  pointer  ♦/ 

/♦  ♦/ 

/♦  Variables  Returned:  verror  -  Integer  pointer  ♦/ 

/♦  misclassifier  -  Integer  array  ♦/ 

/♦  Date:  11  Hov  90  Revision:  1.0  ♦/ 

void  test. the  .net  work  (struct  data  ♦data.recordC], 

struct  Hode.data  ♦node.recordC] , 

int  nodes.in.layer[] , 

int  starting.node.in_laycrC] , 

int  number.of .layers , 

int  record, 

int  total.nodes, 

float  class.threshold, 

int  misclassified [] , 

int  verror) 

{ 

int  network.class  =  Oj 
int  error.buffer; 
error.buffer  =  ♦error; 

calculate.feed.forward.network.output(data.record, 

node.record, 
number.of .layers , 
nodes. in.layer, 
start ing.node.in.layer, 
record, 
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total.nodes) ; 


determlne.class.as.largest(node^record, 

nodes. in^layer, 
s t art ing_node_in.lay er , 
tne twork.cXass  4 
number.of .layers , 
class.threshold) ; 


update.errorsCdata.record, 

record, 

netvork.class, 
misclassified, 
terror.buff er) ; 

♦error  =  err or. buffer; 

> 

/^^^^^^^^^^^^^^^^^^^^^^^^♦^^^^(♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦/ 


/♦  Function  lame:  detenaine.class.as.largest  lumber:  2.2  ♦/ 
/♦  Description:  This  function  determines  the  class  as  the  largest  ♦/ 
/♦  output  node  if  the  output  is  above  some  level  ♦/ 
/*  ♦/ 
/♦  Functions  Called:  lone  ♦/ 
/♦  Variables  Passed  In:  lode.record  Structure  array  ♦/ 
/♦  nodes.in.layer  -  Integer  array  ♦/ 
/♦  starting.node.in.layer  ~  Integer  array  ♦/ 
/♦  ♦netsork.class  -  Integer  pointer  ♦/ 
/*  last.layer  -  Integer  ♦/ 
/*  class.threshold  -  Float  ♦/ 
/*  */ 
/♦  Variables  Returned:  ♦network.class  -  Integer  pointer  ♦/ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


void  determine.class.as.largest (struct  Hode.dat a  ♦node.record[] , 

int  nodes.in.layerC] , 

int  starting.node.in.layer[]  , 

int  ♦netuork.class, 

int  last.layer, 

float  class.threshold) 


{ 

int  X,  last.layer.node; 
float  largest  =  0; 

for  (x  =  0;  X  <  nodes.in.layer [last .layer] ;  x++) 

{ 

last.layer.node  =  starting.node.in.layer [last .layer]  +  x; 
if  ((node.record[last.layer_node]->output  >  largest) 

ftk  (node.re cord [last.layer.node] “>output  >  class.threshold)) 

largest  =  node.record[last.layer.node]->output; 

♦network.class  =  x  +  1; 

> 

} 

} 

/tt^04t***t***^************tt**t*******************t***t¥*t*************/ 


/♦  Function  Same:  update.errors  lumber:  2.3  ♦/ 
/♦  Description:  This  function  compares  the  actual  class  of  the  ♦/ 
/♦  data  record  with  the  network  class.  If  the  two  ♦/ 
/♦  are  not  the  same  an  error  is  recorded  and  the  ♦/ 
/♦  record  number  of  the  data  record  is  stored.  ♦/ 
/♦  ♦/ 
/♦  Functions  Called:  Hone  ♦/ 
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/*  Variables  Passed  In:  training  or  test.dat a  -  Structure  array  ♦/ 
/♦  record  -  Integer  */ 
/♦  netRork.class  -  Integer  ♦/ 
/*  misclassified  -  Integer  array  ♦/ 
/♦  ♦class.error  -  Integer  pointer  ♦/ 
/*  ♦/ 
/♦  Variables  Returned:  ♦class.error  -  Integer  pointer  ♦/ 
/*  nisclassifier  -  Integer  array  ♦/ 
/*  Date:  11  lov  90  Revision:  1.0  ♦/ 


void  update^errors (struct  data  ♦data.recordf] » 
int  record, 
int  netvork.class, 
int  misclassifiedC] , 
int  ♦class.error) 

{ 

int  x; 

X  =  ♦class^error; 

if (data.record [record] ->class  !=  netvork.class) 

misclassifiedCx]  =  data^record[record]->nujnber; 
♦class.error  =  ♦class_error  +  1; 

} 

} 


G.3  NETTRAIN 


/♦  Module  VametVETTRAIH  Humber: 3.0  ♦/ 
/*  Description:  This  module  contains  the  functions  necessary  to  ♦/ 
/*  establish  the  netvork  parameters.  */ 
/♦  Modules  Called:  HETOUT,  HETAUX,  HETMATH,  HETSHOW  */ 
i*  Functions  Contained:  3.1  global^HSE.mlniinization  */ 
/♦  3.2  make_nodes_at.dat appoints  ♦/ 
/♦  3.3  center.veights.at.class^averages  ♦/ 
/♦  3.4  k_means_cluster  ♦/ 
/♦  3,5  train^via^kohonen  ♦/ 
/♦  3.6  MSE^rcmainiiig^layers  ♦/ 
/♦  3.7  CE.remaining^layers  ♦/ 
/♦  3.8  CFM.remaining_lay6rs  ♦/ 
/♦  3.9  PHI^last^layer  ♦/ 
/*  3.10  scale.sigmas.by_class_interference  ♦/ 
/♦  3.11  set^sigmas.to.constant  ♦/ 
/♦  3.12  set_sigma_at«P«neighbor^avg  ♦/ 
/♦  ♦/ 
/♦  Date:10  Hov  90  Revision:  1.0  ♦/ 


tinclude  "netvrble .h" 
tinclude  "netfnctn.h" 

FILE  ♦fileptr; 
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/♦  Function  lame:  global.MSE^minimization  lumber:  3.1  */ 

/«  Description:  This  function  performs  a  global  minimization  of  */ 

/*  the  USE  to  find  netuork  weights.  The  equation  is  */ 

/♦  W  ~  [H(transpose)] (inverse) *Y (transpose) «S  */ 

/♦  ♦/ 

/*  Functions  Called:  8,1  determine^Y^matrix  ♦/ 

/♦  8.2  determiners .matrix  ♦/ 

/♦  9.1  make.identity.matrix  */ 

/*  8.3  detenainerK^^matrix  ♦/ 

9.2  dot ermine jnatrix.transpose  */ 

/*  9.3  invert .a^matrix  ♦/ 

/♦  8.4  calculato.w  eight  .matrix  ♦/ 

/*  ♦/ 

/*  Variables  Passed  In:  Training  Data  -  Structure  ♦/ 

/♦  lode  Record  -  Structure  ♦/ 

/♦  nodes  in  layer  "  integer  array  ♦/ 

/♦  starting  node  in  layer  -  integer  array  ♦/ 

/♦  total  nodes  -  integer  ♦/ 

/♦  current  layer  -  integer  ♦/ 

/♦  ♦/ 

/♦  Variables  Returned:  lode  Record  -  Structure  ♦/ 

/♦  Date:  10  lov  90  Revision:  1.0  ♦/ 


void  global.KSEjninimization  (struct  data  ^training.dataC] , 

struct  lode  .data  ♦lode.recordD , 

int  train.set, 

int  nodes.in.layerQ , 

int  starting.node.in.laycrQ , 

int  total.nodes, 

int  current.layer) 

int  row,  nodes; 

float  MTCTRAI1.SET3ETRAI1.SET],  ICTRAII.SETICTRAII.SET], 
YfJRAIl.SET] ETRAII.SET] ,  MCTRAIH.SET] [TRAIl.SET] , 
weight  [TRAIl.SET]  [aASSES]  ,  S  [TRAIl.SET]  [CLASSES] ; 

float  ♦MTptr [TRAIl.SET],  ♦Iptr [TRAIl.SET] , 

♦Yptr [TRAIl.SET] ,  ♦Mptr [TRAIl.SET] , 

♦weightptr [TRAIl.SET],  ♦Sptr [TRAIl.SET] ; 

for  (row  =  0;  row  <  train.set;  row++) 

{ 

Iptr [row]  =  A1 [row] [0] ; 

MTptrCrow]  =  AMT[row][0]; 

YptrCrow]  =  AY[row][0]; 

Mptr  [row]  =  AM[row][0]; 

SptrCrow]  =  AS [row] [0]; 
weightptr [row]  =  Aweight[row] [0] ; 

} 

detcrmine.Y.matrix (lode.record, 

training.dat a, 

train.set, 

nodes.in.layer, 

starting.node.in.layer, 

total.nodes, 

Yptr, 

current.layer) ; 

nodes  =  nodes.in.layer [current.layer-l] ; 
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determine.S.matrix (tr aining.dat a , 
train.set, 
nodes^in.layor , 

Sptr, 

current.layer) ; 

nodes  «  node s^in^layer [current flayer] ; 

nodes  =  nodes.in.layerCcurrent.layer-l] ; 

aake.identity^matrix  (Sptr, 

nodes) ; 

dcternine_K_matrix(  Yptr, 

Mptr, 

nodes^in.layer, 
train^set, 
current.layer) ; 

dctenuine.icatrix.transpose  (MTptr , 

Mptr, 
nodes) ; 


invert _a_aatrix  (Mptr, 

Kptr, 
nodes) ; 

calculate.weight«natrix(Hode_record, 

weightptr, 

Iptr, 

Yptr, 

Sptr, 

nodes.in^lay er , 
starting_node«in^layer, 
train.set , 
current .layer) ; 

} 

/***t******************  End  Global.MSH.Mininization  ♦♦♦♦♦♦♦♦♦*♦♦/ 

/♦Function  Hame:  Mako-nodes.at.dat appoints  Number :3. 2  ♦/ 

/_♦  Description:  This  module  sets  the  Heights  equal  to  the  exemplar  ♦/ 

/♦  values.  The  equation  is  h(1)  =  x(l)  ♦/ 

/♦  ♦/ 

/♦  Functions  Called:  7.2  calculate.layer.O. output  ♦/ 

/♦  7.3  calculato.layer.l.output  ♦/ 

/*  »/ 

/♦  Variables  Passed  In:  Training  Data  -  Structure  ♦/ 

/♦  Node  Data  -  Structure  ♦/ 

/♦  Train.set  -  Integer  ♦/ 

/♦  Hodes.in.layer  -  Integer  array  ♦/ 

/♦  Sigma^Hax  -  float  ♦/ 

/♦  Output  .Max  -  float  ♦/ 

/♦  Start ing«Bode.in.layer  -  Integer  array  ♦/ 

/♦  Total.Hodes  -  Integer  ♦/ 

/♦  Nodes.l  -  Integer  ♦/ 

/♦  */ 

/♦  Variables  Returned:  Node.record  -  Structure  ♦/ 

/*  ♦/ 

/♦  Date: 10  Nov  90  Revision: 1.0  ♦/ 
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void  aake^nodes.at.data.poijitsCstruct  data  '^data.racord[3 , 

struct;  Hode^data  ♦n.record[], 

int  record.no, 

int  nodes_in_layer[] , 

float  sigma.mar, 

float  out put .max, 

int  start ing.node.in.layer □  , 

int  total.nodes, 

int  nodes.l) 


{ 


int  record,  y,  nodel,  covered,  current.node,  nev.node.number,  xfer.f unction; 
record  =  0; 

nodes.in.layerC']  =  0; 

for  (record  =*  0;  record  <  record.no;  record++) 

{ 

if(nodes.in.layerCl3  <  nodes.l  ) 
covered  =  0; 

calculate.layer.O.output  (data.record, 
n.record, 
nodes.in.laycr, 
record) ; 

calculate.layer.l.output  (data.record, 
n.record, 
nodes. in.layer, 
start ing.node.in.lay er , 
total.nodes) ; 


for  (nodel  =  0;  nodel  <  nodes.in.layer[l] ;  nodel++) 

current.node  =  starting^node.in.layer[l3  +  nodel; 
if  ((n.record  [current .node3*>output  >  output.max) 

kk 

(n.record[current.node3“>class  —  data.record [record3->class)) 
covered  =  covered  +1; 

} 

if  (covered  ==  0) 

new.node.number  =  starting.node.in.layor[l3  +  nodcs.in.layer[l3 ; 
for  (y  =  0;  y  <  nodes.in.layer[03;  y++) 

{ 

n.record  [new.node.nuiaber3->weightCy3  =  data.record  [record3“>vectorCy3 ; 
n.record  [new.nodo.number3->sigma[y3  =  sigina_inax; 

} 

n.record  [ncv.node.nuinbcr3“>cla8S  =  data.record  [record3“>class; 
nodes.in.layer[l3  =  nodes.in.layer[l3  +1; 

} 

} 

} 

} 

/^:tr**************  End  MaXe  Hodes  at  Data  Points'>*****>>/ 


/*  Function  ffame:  Center.tfeights.at.Class.Avcragcs  iiuiQbcr:3.3  */ 
/♦  Description:  This  function  sets  the  node  weights  equal  to  the  */ 
/♦  averages  of  clusters  of  the  same  class  */ 
/♦  w(+)  =  v(-)  +  [x-w(-)3/(M+l)  ♦/ 
/♦  ♦/ 
/*  Functions  Called;  9.4  update. aver age  */ 
/♦  */ 
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/♦  Variables  Passed  In;  Training  data  -  Structure  ♦/ 
/♦  Hode.record  -  Structure  */ 
/♦  rocord.no  -  integer  ♦/ 
/*  nodes. in.layer  -  integer  array  ♦/ 
/♦  average.threshold  *-  float  ♦/ 
/♦  sigma.threshold  -  float  ♦/ 
/*  start ing.node.in.lay0r  -  integer  array  ♦/ 
/♦  total.nodes  ~  integer  ♦/ 
/♦  current.layer  -  integer  ♦/ 
/♦  nodes.l.iufxiiaum  -  integer  ♦/ 
/*  ♦/ 
/*  Variables  Returned:  Sode.record  -  structure  ♦/ 
/♦  ♦/ 
/♦  DaterlO  Kov  90  Revision:  1.0  ♦/ 


void  centcr_weights.at.class.averages(struct  data  ♦data.recordD , 

struct  lode.data  ♦node.record[] » 

int  record.no, 

int  nodes_in.lay^rC] , 

float  threshold, 

float  sigi&a.max, 

int  starting_node.in_layorO , 

int  tctal.nodes, 

int  curi‘cr»**.layer, 

int  nodes.nax) 

{ 

int  nearest .node  =  0; 
int  iteration; 
double  min.distance  =  1000; 
double  distance,  buffer; 
double  exponent.l  =»  2; 
double  exponent.2  =  .5; 
int  new.node  ~  0; 
float  neu.average; 

int  record,  y,  x,  covered,  elements  [TRAIJf.SET+TEST.SET] ,  current  .node,  previous.layer.node,  neu.node.number 
nodes.in.layGr [current .layer]  =  0; 
record  =  0; 


for  (x=0;  X  <  TRAIH.SET  +  TEST.SET;  x++) 
elements  [x]s=0; 
do 
{ 

new.node  =  0; 

for  (record  =  0;  record  <  record.no;  record++) 

{ 

if  (nodes.in.laycr  [current .layer]  <  nodes.max) 

{ 

min.distance  =  1000; 
covered  =  0; 

calculate.layor_0_output(data.rGcord, 
node.record, 
nodes. in.layer, 
record) ; 

for  (x  =  0;  X  <  nodes_in.layer[current.layer] ;  x++) 

{ 

current.node  *=  start  ing.node.  in.layer  [current .layer]  +  x; 
if  (node.record [current .node] ->class  ==  data.record [record] ->class) 

{ 

buffer  =  0; 

for  (y  =  0;  y  <  nodes.in.layer [current .layer-1] ;  y++) 

{ 

previous.layer.node  =  starting.node.in.layer[currGnt.layer-l]+y; 
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distance  =  node.record  [current  .node] ->wQight[previous.layor«node] 
-node.record [pre vious.layer.node] ->output ; 

'  distance  =  pov(distance,  ezponent.l); 
buffer  B  buffer  +  distance; 

} 

distance  »  pov (buffer, exponent. 2) ; 
if  (distance  <  min.distance) 

{ 

neare8t.node  =  current.node ; 
min.distance  ~  distance; 

> 

} 

> 

if  (min.distance  <  threshold) 

{ 

X  =  nearest.nodo-starting.node.in.layer [current. layer] ; 
elements [x]  «  elements [x]+l; 

for  (  y  =  0;  y  <  node8.in.layer [current . layer~l] ;  y++) 

previous.layer.node  =  start ing.node.in.layer[current.layer“l]  +y; 
update.average(node.record[nearest.node]^>Height[prcvious.layer.node] , 
elements [x] , 

node.record [previous.layer.nodo] “>output , 
tnev.average) ; 

node.record [nearest.node]~>veight [previous.layer.node]  =  new.average; 

} 

X  =  node 8. in.layer [current .layer]  ; 
covered  =  covered  +1; 


else 

{ 

nev.node.number  =  start ing.node.in.layer [current .layer]  +  x; 
for  Cy  =  0;  y  <  nodcs.in.layer [current. lay or-1] ;  y++) 

{ 

previous.layer.node  =  start ing.node.in.layer [current .layer-1]  +  y; 

node.record [neH.node.number]->weight [previous.layer.node]  =  node.record  [previous.layer.node] ->output ; 
node.record [ne».node.number] ->sigma [previous.layer.node]  ~  s igma.maz ; 

'i 

node.record [neff.nod6.number]->class  =  data.record [record] ->class; 
elements [x]  =  1; 

nodes. in.layer  [current  .layer]  =  nodes.in.layer[current.layer]+l ; 

/♦  printf  ("\ncr eat ed  node  y,d'*,uew.nod8.number) ;  */ 

nev.node  =  1; 

} 

} 

> 

} 

while  (new.node  ==  1); 

} 

/:^*****:^*t4***t*ETid  Center  Weights  at  Class  Centers  ♦♦♦*♦♦*/ 


/*  Function  Iame:K-Keans  Cluster  lumber: 3. 4  »/ 
/*  Description:  This  function  implements  the  K-Neans  Clustering  */ 
/♦  algorithm  to  set  the  weights.  ♦/ 
/*  w(+)  =  (l/Dsum  [x(n)]  */ 
/*  */ 
/♦  Functions  Called:  7.2  calculate.layer.O.output  ♦/ 
/♦  8.28  find.nearest.neighbor  ♦/ 
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/♦  9.4  update.average  ♦/ 
/♦  */ 
/♦  Variables  Passed  In:  Training  Data  -  Structure  ♦/ 
/♦  lode.record  -  Structure  ♦/ 
/♦  Train.sot  -  Structure  ♦/ 
/♦  lodes.i.MaziiBum  Integer  ♦/ 
/♦  lodes^in^Layer  -  Integer  array  ♦/ 
/♦  Starting^Iode^in^layer  ~  Integer  array  ♦/ 
/*  Current^Layer  -  Integer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  Sfode.record  -  Structure  ♦/ 
/*  Date:  10  Mov  90  Revision: 1.0  ♦/ 


void  k.means^cluster (struct  data  edata.recordD  , 

struct  lode.dat a  enode.record[] , 

int  record.no, 

int  number. of .clusters, 

int  nodes. in.layer[], 

int  starting.node.in.layerC] » 

int  current .layer) 

{ 

int  record,  x,  y,  z,  current .node,  nearest. node,  previons.layer.node ,  update; 
int  total.elements,  current.record,  elements; 
float  nev.average,  currcnt.avgCTEST.SET] ; 

int  element [TEST.SET] [TEST.SBT] ,  elements.in_clusterCTEST.SET]; 
double  distance,  nearest  .distance,  buffer,  difference; 
nev.average  =  0; 

for  (record  =  0;  record  <  number. of .clusters ;  record++) 

calculat e.layer.O.output (dat  a.record , 
node.record, 
nodes.in.layer, 
record) ; 

current.node  »  starting.node.in.layer[current_layer]  +  record; 
for  (y  =  0;  y  <  nodes_in.layer[current.layer-l]  ;y++) 

{ 

previous.layer.node  =  st art ing_node_in.layer [current .lay er-lj+y; 

node.record [current. node] "> weight [previous.layer .node]  =  node.record [previous.layer.node] ->output 

> 

> 

do 

{ 

for  (x  =  0;  X  <  number.of. clusters;  x++) 
elements.in.clusterCx]  =  0; 
update  =  0; 

for  (record  =  0;  record  <  record.no;  record++) 

find.nearest.neighbor (dat a.record, 
node.record, 
record, 

nodes.in.layer , 
start ing.node.in.layer , 
current .layer, 
ftnearest.node); 

nearest. node  =  nearest.node  -  starting.node.in.laycr [current .layer] ; 
current .record  =*  clement c.in.cluctcr [nearest.node] ; 
element [nearest.node] [current.record]  =  record; 

elements.in. cluster [nearest.node]  =  elements_in.cluster[ncarest.node]+l ; 

} 

for  (x  =  0;  X  <  nodes.in.layer [current .layer] ;  x++) 

{ 
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curront^node  «  startiiig»node«in.layerCcurrent»layer]  +  x; 
for  (y  =  0;  y  <  nodcs^in.layer [current. layar-1] ;  y++) 
current.aygCy]  =  0; 

total.elements  »  element8.ixi.cluster[x] ; 
for  (z  =  0;  z  <  total.elements;  z++) 

{ 

record  »  element [z]  [z] ; 
elements  »  z  +  1; 

calculate.layer.O.outputCdata.rccord, 
node.record» 
nodes.in.layer, 
record) ; 

for  (y  =  0;  y  <  nodes.in.layer[current_layer-l] ;  y++) 

{ 

preyious.layer.node  *  starting.node.in.layerCcurrent.layer-l]  +  y; 
update.average (current .avg[y] , 
elements, 

node.recordCprevious.layer.node] “>output , 
ftnev.average) ; 

current .aygCy]  »  new.ayerago; 

> 

> 

for  (y  =  0;  y  <  ncdes.in.layer [current. layer-1] ;  y++) 

{ 

preyious.layer.node  =  starting.node.in.layor[current.layer-i]  +  y; 
difference  =  node.re cord [current. node] ->h eight [preyious.layer.node]- current.ayg[y] 
if  (fabs (difference)  >  .00001) 
update  =  1; 

node.rocord[current.node]->weight  [preyious.layer.node]  =  current. ayg[y] ; 

} 

} 

} 

while (update  !=  0); 

> 

/^t^t^************  End  K-Means  Cluster  *******•>************/ 


/* 

Function  lame:  Kohonen  Training  lumber: 3. 5 

*/ 

/*  Description:  This  function  updates  the  weights  via  the  Xohonen 

*/ 

/* 

training  algorithm 

♦/ 

h 

w(+) 

=  w(-)  +  a[x-w(-)] 

*/ 

/* 

♦/ 

/* 

Functions  Called: 

4 . 9  get.random.record 

♦/ 

/* 

7 . 2  calculate.layer.O.output 

♦/ 

/* 

8 . 13  calc.dist.output_to.nxt.lyr 

♦/ 

/* 

8.14  find.nearest.element 

*/ 

/♦ 

8.11  get_linear.training.eta 

*/ 

/* 

8.12  get.kohonen.neighborhood 

*/ 

/* 

8.15  f ind_kohonen.boxindaries 

*/ 

/♦ 

8 . 16  determine.neighborhood.elements 

♦/ 

/* 

8.17  train.kohoncn.weights 

*/ 

h 

♦/ 

!*  Variables  Passed  In:  Training.Data  -  Structure 

*/ 

/* 

lode.Record  -  Structure 

♦/ 

/♦ 

lodes.in.layer  -  integer  array 

♦/ 

/♦ 

Starting.Iode.in.layer  -  integer  array 

/♦ 

leighborhoods  -  integer  array 

*/ 

/* 

Train  Width  -  integer  array 

/* 

Train  Scale  -  float  array 

♦/ 

/♦ 

Width.number  -  integer 

♦/ 

/♦ 

lodes.x  -  integer 

♦/ 

/♦ 

Hodes.y  -  integer 

*/ 
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/*  Current^layer  -  intogcr  ♦/ 
/♦  Train.set  -  integer  */ 
/*  ♦/ 
/*  Variables  Returned:  lode.record  -  Structure  ♦/ 
/*  Date:  10  Iot  90  Revision:  1.0  ♦/ 


void  train.via.kohononCstruct  data  ♦data^recordCJ , 

struct  lode.data  *node.recordG » 

int  nodes^in.layer[3 , 

int  starting.node.in.laycrG « 

int  neighborhoods  □ , 

int  train.uidth[] , 

float  train.scale[] , 

int  width.no, 

int  nodes.x, 

int  nodos.y, 

int  current. layer, 

int  train.set, 

int  kohonen.iterations, 
unsigned  seed) 


{ 

int  X,  y,  current.node,  iterations,  record; 
int  winner.node; 

int  neighbors,  left,  right,  up,  down,  nodes.to.update; 
int  update.node[100] ; 
float  eta; 

float  distance [100] ; 


record  =  0; 
iterations  ~  0; 
srand(seed) ; 
do 
{ 

get.random.record(train.set, 

Rrecord) ; 

calculate.layer.O.output (dat a.record , 
node.record, 
nodes. in.layer, 
record) ; 

calc.dist.outputs.to.nxt.lyr (node.record, 

nodes.in.layer, 
start ing.node.in.layer, 
current .layer, 
distance) ; 


f ind.nearest.element (distance , 

nodes.in.layer Cl] , 
kwlnner.node); 


winner.node  =  starting.node.in.layorCcxirrent.layer] 
+  winner.node; 

gct.lincar.training.eta(train.width, 

train.scale, 

iterations, 

Reta, 

width.no) ; 


get.kohonen.neighborhood(train.width, 

iterations, 
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neighborhoods 

width.no, 

ftneighbors); 


f  ind.kohonen^boundarieH  (e  inner  .node , 

start ing.node. in.lay er , 

current .layer, 

nodes.z, 

nodes.y, 

neighbors, 

Jfcleft, 

tright, 

Jtup, 
ftdovn) ; 

dot ermine.neighborhood_ elements (left , 

right, 

up, 

do«n, 

ftnodes.to.update , 

update.node, 

start ing.node.in.layer , 

nodes.x, 

current .layer) ; 

train.kohonen.9eights(node.record, 

nodes.in.layer, 
st arting_node.in.layer , 
current .layer, 
nodes.to.update , 
update.node, 
eta) ; 

iterations  -  iterations  +  1; 

> 

while  (iterations  <  kohonen.it erat ions) ; 

} 

/t*********  End  train  via  kohohen 

/*  Function  lame:  MSE  Remaining  Layers  lumber :3. 6  */ 

/*  Description:  The  function  performs  backpropagation  to  optimize  ♦/ 
/*  the  MSE  objective  function.  */ 

/♦  ♦/ 

/*  Functions  Called:  4.8  get.random.class.record  ♦/ 

/♦  7.1  calculate.feedforvard.network.output  ♦/ 

/*  8.10  calculate.errors.in.output  ♦/ 

/*  8.5  MSE.last. layer  ♦/ 

/♦  8.8  MSE.mid.layer  ♦/ 

/♦  8.9  MSE.lst. layer  */ 

/♦  S.9  correct.node.weights  */ 

/*  2.1  test.the.network  ♦/ 

/♦  9.6  calculate.percentage  ♦/ 

/*  ♦/ 

/♦  Variables  Passed  In:  lode.Record  -  Structure  ♦/ 

/♦  Training.Data  -  Structure  ♦/ 

/♦  Test.Data  -  Structure  »/ 

/*  Transfer.Function  -  Integer  array  ♦/ 

/*  Hodes.in.Layer  -  Integer  array  ♦/ 

/♦  Starting.Hode.in.Layer  -  Integer  array  */ 

/♦  lumber.of.Layers  -  Integer  ♦/ 

/♦  Current  .Layer  -  Integer  ♦/ 

/*  Train.Set  -  Integer  ♦/ 

/♦  Test.Set  -  Integer  */ 
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/♦ 

HSE.Eta  -  Float 

*/ 

/* 

Total.Iodes  -  Integer 

*/ 

/♦ 

HSE.Successes  -  Integer 

♦/ 

/* 

MSE.Epsilon  -  float 

♦/ 

/♦ 

HSE.Iterations  -  Integer 

♦/ 

/* 

MSE.momentU]n  -  float 

♦/ 

/* 

Classes  integer 

*/ 

/♦ 

Record.Seed  -  unsigned 

♦/ 

/♦ 

File.ptr  -  File  pointer 

*/ 

/* 

♦/ 

/♦  Variables  Returned: 

Hode.Record  -  Structure 

♦/ 

/♦  Date;  10  Jov  90 

Revision:  1.0 

♦/ 

void  HSE.remaining.layors (struct  Vodo.data  ♦node.recordC] , 

struct  data  ♦data^record[] , 
struct  data  ♦test^recordQ , 
int  transfer.functionC] , 
int  nodes.in.layerC] , 
int  starting„node_in_lay©rC] , 
int  numb er^of. layers, 
int  current .layer, 
int  train.set, 
int  test.set, 
float  eta, 

int  total.nodes, 

int  MSE.successes, 
float  epsilon, 

int  backprop.iterations, 
float  alpha, 

int  classes, 
unsigned  seed, 

FILE  ♦file.ptr) 


float  desired.output [CLASSES] ; 

int  X,  y,  error,  record,  layer,  node; 

int  success  =  0; 

int  iteration  ~  0; 

int  error.interval.count  =  0; 

int  error.covnt  =  0; 

int  misclassifiedCTRAll.SET] ; 

int  correct. class  =  0; 

float  class.threshold  =  0; 

float  per.cent. correct  =0.0; 

float  old.wght [TOTAL.IODES] [TOTAL.HODES] ; 

float  ♦ught.ptrCTOTAL.BODES]; 

class.threshold  =  l~epsilon; 

for  (x  =  0;  X  <  total.nodes;  x++) 

{ 

for  (y  =  0;  y  <  total.nodes;  y++) 
old.wght  [x][y]  =  0; 
wght.ptrCx]  =  Aold.wght [x] [0] ; 

} 

srand(seed); 

do 

{ 

error  =  0; 

get.random.class.record(data.record, 
train.set, 
classes, 
^record) ; 
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for  (x  =  0;  X  <  nodes_in_layerCnumber«of_layGrs] ;  x++) 

{ 

if  (x  ==  data.recordCrecord]->class  -1) 
des ired. output Cx3  =  1.00; 
also 

desired^outputCxl  =  0,00; 


calculate.fe6d«forward.network.output(data.record, 

node.xecord» 

number.of .layers , 

node8.in_layer> 

starting_node.in.layer, 

record, 

total.nodes) ; 

calculate.errors_iii.output(node.record, 

dosired.output , 
nodes.in.layer, 

St  art ing_node.in.layer , 
number.of .layers , 
terror, 

epsilon) ; 


if  (error  !=  0) 

{ 

success  =  0; 

for  (layer  =  number.of .layers ;  layer  >  current .layer  -1;  layer — ) 

if  (layer  ==  number_of.layers) 

HSE.last. layer (node.record , 

desired.output , 

nodes.in.layer, 

starting.node.in.layer, 

layer, 

eta, 

epsilon, 

wght.ptr, 
alpha) ; 

else  if  (layer  ==  number.of.layers-l) 

MSE.mid.layer (node.record, 

desired.output , 

nodes.in.layer, 

starting.node.in.layer, 

layer, 

eta, 

wght.ptr, 
alpha) ; 

else  if  (layer  ==  number. of .layers -2) 

MSE.l St .layer (node.record , 

desired.output , 

nodes.in.layer , 

starting.node.in.layer, 

layer, 

eta, 

wght.ptr, 
alpha) ; 

} 

} 

else 

success  =  success  +  1; 

iteration  =  iteration  +1; 
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correct.node«weights(iiode„record, 

total.nodes) ; 

error.interval.count  =  error_interval_count  +  1; 
if  (crror.interval^count  ==  1000) 

{ 

error.count  =  0; 
for  (x  =  0;  X  <  train.set;  x++) 
test«the«network(data.record, 
node.record, 
nodes^in^layor, 
starting.node^in.layer, 
nuinber.of  .layers , 

x» 

total.nodes, 

class.threshold, 

inisclassified> 

kerror.count); 

correct.class  =  train.set  -  error.count; 

calculat e.percentage ( (float) correct_class , 

(float) train.set , 
tper.cent.correct) ; 

fprintf(file.ptr,"\niteration  =  y.d  training  correct  =  y.f", 
iteration,  per.cent.correct) ; 


error. count  =  0; 
for  (x  =  0;  X  <  test. sot;  x++) 
test.the.networlc(test. record, 

node.record, 
nodes.in.layer, 
start ing.node.in.layer, 
number.of .layers , 

ac, 

total.nodes, 
class.threshold, 
misclassified, 
terror.count) ; 

correct. class  =  test.set  ~  error.count; 
calculate.percentage ( (float) correct.class , 

(float)test.set, 
ftper.cent.correct) ; 

fprintf  (filo.ptr,"  test  percent  =  '/.f, per.cent.correct)  ; 
error.interval.count  -  0; 

> 

} 

while((iteration  <  backprop.iterations)  kt  (success  <  MSE.successes)) ; 

} 


/,^^^^^**t*******’¥****w**  End  MSE  Remaining  Layers  ♦♦♦♦♦♦♦♦♦♦/ 


/♦  Function  Same;  CE.Remaining.Layers  Humber: 3. 7  ♦/ 
/♦  Description:  The  ftinction  sets  parameters  by  backpropagation  */ 
/♦  according  to  the  CE  objective  function  */ 

/*  */ 

/*  Functions  Galled:  4.8  get.randos.class.record  */ 
/♦  7.1  calculate.f eedf orward.network.output  ♦/ 
/♦  8.10  calculate.errors.in.output  ♦/ 
/♦  8.20  CE.last .layer  */ 
/♦  8.21  CE.mid.layer  ♦/ 
/♦  8.22  CE.first .layer  ♦/ 
/♦  5.9  correct. node.weights  ♦/ 
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/♦  2.1  test.the.letvork  ♦/ 

/♦  9.6  calculate_perc«ntage  ♦/ 

/*  ♦/ 

/*  Variables  Passed  In:  lode.Record  '*  Structure 

/♦  Training^Data  -  Structure  */ 

/♦  Test. Data  -  Structure  ♦/ 

/♦  lodes.ln.Layer  -  Integer  array  ♦/ 

/♦  lumber.of.Layers  -  Integer  ♦/ 

/♦  Start ing.Vode.in.Layer  -  Integer  ♦/ 

/♦  Current .Layer  -  Integer  */ 

/♦  Train.Set  -  Integer  ♦/ 

/♦  Test.Set  -  Integer  ♦/ 

/♦  CE.eta  -  Float  */ 

/♦  Total.Iodes  -  Integer  */ 

/*  CE.Successes  -  Integer  ♦/ 

/♦  CE.Epsilon  -  Float  ♦/ 

/♦  CE.Iterations  -  Integer  ♦/ 

/♦  CE.Homentun  -  Float  ♦/ 

/♦  Classes  -  Integer  ♦/ 

/♦  Record.Seed  •*  Unsigned  ♦/ 

/♦  File.Ptr  -  File  pointer  ♦/ 

/♦  ♦/ 

/*  Variables  Returned:  Hode.Record  -  Structure  ♦/ 

/♦  Date: 10  Iot  90  Revision:  1.0  ♦/ 

void  CE.remaining.layers (struct  lode.data  ♦node.recordQ » 
struct  data  ♦data.recordC] , 
struct  data  *test.record[] , 
int  nodes.in.layerC] , 
int  starting.node.in.layer[] , 
int  number.of. layers, 
int  current .lay or, 
int  train.set, 
int  test.set, 
float  eta, 
int  total.nodes, 
int  CE.successes, 
float  epsilon, 
int  CE.it orations, 
float  momentum, 
int  classes, 
unsigned  seed, 

FILE  efile.ptr) 

float  desired. output [CLASSES] ; 

int  X,  y,  error,  record,  layer,  node; 

int  success  =  0; 

int  iteration  =  0; 

int  erxor.interval.count  =  0; 

int  error.count  =  0; 

int  misclassifiedCTRAII.SET] ; 

int  correct .class  =  0; 

float  per.ccnt.correct  =  0.0; 

float  old.wght [TOTAL.IODES] [TOTAL.IODES] ; 

float  ♦wght.ptr[TOTAL.IODES]; 

float  class.threshold  =  1 -epsilon; 

float  ncw.eta  =  eta/(nodes.in.layer[number.of.layers]*2.3) ; 

for  (x  =  0;  X  <  total.nodes;  x++) 

{ 

for  (y  =  0;  y  <  total.node3;  y++) 
old.wght [x] [y]  =  0; 
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wght.ptrCx]  =  Aold^wght [x] [0] ; 

> 

srandCseod); 

do 

{ 

error  =  0; 

get.random^class^recordCdata^rocord, 

train.set> 

classes, 

^record) ; 

for  (x  =  0;  X  <  nodes»in«layer [immber.of flayers] ;  x++) 

{ 

if  (x  ==  data^record [record] “>class-l) 
desired.output[x]  =  1.00; 
else 

desired.otttput  [x]  =  0.00; 

> 

calculate.f eed^f orvard^neteork^output (data.record , 

node.record, 
numbcr^of flayers , 
nodes^in^layer , 
start ing.node_ in«lay er , 
record, 
total.nodes) ; 

calculate.errors_in.output (node^record, 

desired^output , 
nodes.in.layer, 
starting.node«iii«layer, 
nmnber^of^layers , 
terror , 
epsilon) ; 


if  (error  !=  0) 

i 

success  =  0; 

for  (layer  =  number.of .layers ;  layer  >  current .layer  -1;  layer — ) 

{ 

if  (layer  ==  nuaber.of.layers) 

CE.last .layer (node_record , 

nodes.in.layer , 

start ing_node.in.lay er , 

layer, 

wght.ptr, 

new.eta, 

momentum, 

desired.output) ; 

else  if  (layer  ==  number. of  .layers  -1) 
CE_mid.layer(node.record, 

nodes.in.layer, 

start ing.node.in.layer, 

layer, 

wght.ptr, 

new.eta, 

momentum, 

desired.output) ; 

else  if  (layer  ==  number_of.layers-2) 
CE.first.layer(node.record, 

nodes.in.layor, 
start ing.node.in.layer, 
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layer, 

¥ght„ptr, 

nev.eta, 

momentum, 

'  desired. output, 

total.nodes); 

} 

> 

else 

success  “  success  +  1; 
iteration  «  iteration  +  1 ; 
correct.node.neights(node.record, 
total.nodos) ; 

error.  ittterval.count  =  error.interval.count  +  1; 
if  (error.interTal.count  ==  1000) 

{ 

error.count  =  0; 

for  (r  a  0;  x  <  train.set;  x++) 
test.the.netvork(data.r6cord, 
node.record, 
nodes.in.layer, 
start ing.nodo.in.layer, 
number. of .layers , 

x> 

total.nodes, 
clas  s.thre  shold , 
misclassified, 
terror .count) ; 

correct.class  =  train.set  -  error.count; 

calculate.percentageC (float) correct .class, 

(float) train.set , 
tper.cent. correct) ; 

fprintf(file.ptr,'*',niteration  »  y,d  training  correct  =  %f*', 
iteration,  per. cent .correct) ; 


error.count  =  0; 
for  (x  =  0;  X  <  test. set;  x++) 
test.the.netBork(test.record , 
node.record, 
node8.in.layer, 
start ing.node.in.layer , 
number.of .layers , 

X, 

total.nodes, 
class.threshold, 
misclassified, 
t error.count) ; 

correct.class  =  test.set  -  error.count; 
calculat  e.per centage ( (float ) correct.class , 

(float) test.set, 
kper.cent.correct) ; 

fprintf (file.ptr,**  test  correct  =  y.f",pcr.ccnt. correct); 
error.interval.count  =  0; 

} 

} 

while ((iteration  <  CE.iterations)  tk  (success  <  CE.successes)) ; 

} 

/0044it******0*************4***0********4i***********4*>^*****/ 

/♦  End  CE  Remaining  Layers  ♦/ 
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/• 

Function  lame:  CFK.Remaining.Layers  Iumber:3.8 

♦/ 

/♦  Description: 

The  function  sets  parameters  by  bachpropagation 

♦/ 

/* 

according  to  the  CE  objective  ’.function 

♦/ 

/*  Functions  Called:  4.8  gct.random.class.rocord 

♦/ 

/* 

7 . 1  calculate.f eedforuard.net«ork.output 

*/ 

h 

2.2  determine.class.as.largest 

*/ 

h 

8 . 26  find.second.highest.node 

♦/ 

h 

8.23  calculate.zn 

*/ 

/♦ 

8.20  CFM.last.layer 

*/ 

h 

8.21  CFM.mid.layer 

♦/ 

/* 

8.22  CFM.first.layer 

*/ 

/* 

5.9  correct.node.veights 

*/ 

/* 

2.1  test.the.netvoxk 

*/ 

/* 

9.6  calculate.per cent age 

♦/ 

/♦ 

*/ 

/♦  Variables  Passed  In:  lode.Record  ~  Structure 

♦/ 

/♦ 

Training.Data  -  Structure 

♦/ 

/* 

Test.Data  -  Structure 

♦/ 

/♦ 

lodes. in.Laycr  -  Integer  2irray 

♦/ 

/♦ 

Starting.Iode.in.layer  ~  Integer  array 

*/ 

/♦ 

Sumber. of .Layers  -  Integer 

♦/ 

/♦ 

Current.Layer  -  Integer 

♦/ 

h 

Train.Set  -  Integer 

*/ 

/* 

Test.Set  -  Integer 

*/ 

/* 

CE.eta  -  Float 

*/ 

/♦ 

Total.Hodes  -  Integer 

*/ 

/♦ 

CFH.Successes  -  Integer 

♦/ 

h 

CFH.Iterations  -  Integer 

•/ 

h 

CFM. alpha  -  Float 

*/ 

/* 

CFM.beta  -  Float 

♦/ 

/♦ 

CFm.zeta  ~  Float 

*/ 

/♦ 

CE.Homentum  -  Float 

♦/ 

/* 

CE.delat  -  Float 

♦/ 

/* 

Classes  ^  Integer 

*/ 

/♦ 

Record.Seed  -  Unsigned 

♦/ 

/♦ 

File.Ptr  ~  File  pointer 

♦/ 

/♦ 

♦/ 

/♦ 

Variables  Returned:  Hode.Record  -  Structure 

*/ 

/*  Date:  10  lov 

90  Revision:  1.0 

*/ 

/*****0***^t***********0**************t************9****************/ 


/♦  CFM  Remaining  Lyrs  ♦/ 


void  CFM.rcmaining^layers (struct  Hodo.data  ♦node^recordC] , 
struct  data  ♦data.recordC] , 
struct  data  *test.recordn , 
int  nodes_in«layerC] , 
int  starting«node_in_layer[], 
int  numb er.of .layers, 
int  current.layer, 
int  train.sot, 
int  test.set, 
float  eta, 
int  total.nodes, 
int  CFM.succcsses, 
int  CFM.iterations, 
float  alpha. 
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float  beta, 
float  zeta, 
float  momentum, 
float  delta, 
int  classes, 
unsigned  seed, 
FILE  ♦f ile.ptr) 


int  X,  y,  record,  layer,  correct. node,  winner.node; 

int  netvork.class  ^  0; 

int  success  »  0; 

int  iteration  =  0; 

int  error.intorval.count  =  0; 

int  error.count  »  0; 

int  misclassif ied[TlUII.S£T] ; 

int  correct. class  =  0; 

float  per. cent. correct  =  0; 

int  next.highest.node; 

float  class.threshold  ^  0; 

float  new.eta  »  0; 

float  znCCLASSES] ; 

float  old.wghtCTOTAL.IODESlCTOTAL.IODES] ; 
float  ♦wght.ptrCTOTAL.IODES]; 
float  epsilon  =  .9; 

for  (x  =0;  X  <  total.nodes;  x++) 

{ 

for  (y  =  0;  y  <  total.nodes;  y++) 
old.ught [x] Cy]  =  0; 
wght.ptrCx]  =  told.ught [x] [0] ; 

} 

nev.eta  =  eta  ♦  beta  ♦  alpha/ (nodes.in.layerCnumber_of. layers] ~1 ) ; 
srandCseed) ; 

do 

{ 

iteration  =  iteration  +  1; 

get.random.clas8.record(data.record, 

train.set, 

classes, 

Arecord) ; 

calculate.feed.forvard.nQtvork.output (data.record, 

node.record, 

number. of .layers , 

nodes.in.layer, 

starting.node.in.layer, 

record, 

total.nodes) ; 

correct. node  =  starting.node.in.layer [number.of. layers] 

+  data.recordCrecord]->class  -  1; 


deteimine.class.as.largest (node.record, 

nodes.in.layer, 
st arting.node.in.layer , 
Anetvork.class, 
number.of .layers , 
class.threshold) ; 

winner. node  =  start ing.node.in.layer [number.of .layers] 
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+  netvork.class  •  1; 


if  (vinner.node  ==  correct^node) 

find.second.highest.node  (node.record, 

nodes.in.lnyer, 
start ing.node.in.layer , 
number.of .layers , 
dinner. node, 
Jtnext.highest.node) ; 


if  ((winner.node  !»  correct.node) 

1 1  (node.record [correct.node] ->output 

~  node.recordCnext.highest.node]->output  <  delta)) 

{ 

success  =  0; 

calculate.zn  (node.record, 

nodes.in.layer, 

starting.node.in.layer, 

number. of .layers , 

correct.node , 

zn, 

beta, 

zeta) ; 

for  (layer  =  number.of. layers;  layer  >  current .layer-1 ;  layer — ) 

{ 

if  (layer  ==  number.of.layers) 

CFM.last. layer (node .record, 

nodes.in.layer, 

start ing.node.in.layer , 

layer, 

zn, 

correct.node , 
neu.eta, 
sght.ptr, 
momentum) ; 

else  if  (layer  ==  number.of. layers-1) 
CFM.mid.layer(uode.record, 

nodes.in.layer , 

starting.node.in.layer, 

layer, 

zn, 

correct.node, 
new.eta, 
wght.ptr, 
momentum) ; 

else  if  (layer  ==  number.of.layers-2) 

CFM.first. layer (node.record, 

nodes.in.layer, 
starting.node.in.layer , 
layer, 
zn, 

correct.node, 
new.eta, 
wght.ptr, 
momentiim, 
total.nodes) ; 

} 

} 

else 

success  =  success  +  1; 
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error.interval.count  =  error_interval«count  +  1; 
if  (error.interval^count  —  1000) 

{ 

error.count  «  0; 
for  (x  =  0;  X  <  train«sot;  x++) 
test»the_networX(data^r«cord, 
node.record, 
nodcs^in.layer, 
starting^node.in.laycr, 
ntunber^of  .layers , 

3C* 

total.nodes^ 
epsilon, 
jnisclassified, 
terror.count) ; 

correct.class  =  train^set  -  error.count; 

calculat  e.percent age ( (float) correct.class , 

(float) train.set , 
tper.cent .correct) ; 

fprintf  (file.ptr,‘*\niterations  =  Xd  training  correct  =  51f‘'> 
iteration,  per.cent.correct) ; 


error.count  =  0; 
for  (x  =  0;  X  <  test.set;  x++) 
test.the.netHork(test.record, 
node.record, 
nodcs.in.layer, 
start  ing.node.iji.layer, 
number.of .layers , 

X, 

total.nodes, 
epsilon, 
misclassified, 
terror.connt) ; 

correct.class  =  test.set  -  error.count; 
calculate_percentag6( (float) correct.class , 

(float) test.set, 
ftper.cent.correct) ; 

fprintf (file.pt r,"  test  correct  =  )tf'‘,per.cent.correct) ; 
error.interval.count  =  0; 

} 

} 

while (success  <  CFM.successes  kk  iteration  <  CFH.iterations) ; 

} 


/*t*******************************t*******t***i^***********/ 

/♦  End  CFM  Remaining  Lyrs  ♦/ 

/^^i^:^0tl^******i^***t***************************************/ 

^^I^^t^^***************t*****************************t****/ 

/*  Function  NameiPIH.last.layer  Humber: 3. 9  ♦/ 

/♦  Description:  This  function  sets  the  network  weights  in  the  ♦/ 

/*  output  layer  equal  to  1  and  connects  the  output  */ 

/*  layer  nodes  only  to  the  nodes  of  the  same  class  */ 

/♦  in  the  hidden  layer  ♦/ 

/*  ♦/ 
/♦  Functions  Called:  Hone  ♦/ 

/♦  Variables  Passed  In:  Hode.Record  -  Structure  ♦/ 

/>»  Hodes.in.Layer  -  Integer  array  ♦/ 

/♦  Starting.Hode.in.Layer  -  Integer  array  */ 

/♦  Current .Layer  -  Integer  */ 
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/*  ♦/ 
/♦  Variables  Returned:  lode.Record  -  Structure  ♦/ 

/♦  Date: 10  loy  90  ReTision:1.0  ♦/ 


Toid  Pll.last.layerCstruct  lode.data  enode.recordCr « 

int  nodes.in^layerC] » 

int  starting.node.in.layer[] , 

int  current .layer) 


{ 

int  X,  y,  current.node ,  previous.lyr.node; 
float  node s.of. class; 

for  (x  »  0;  X  <  nodes.in.layer[current.layer] ;  x++) 

{ 

current.node  =  starting.node.in_layerCcurrcnt.layer3+x; 
node.record[current_node]“>clas8  =  x  +  1; 
nodes. of. class  ~  0; 

for  (y  =  0;  y  <  nodes.in.layerCcurrent.layer-l] ;  y++) 

{ 

previous.lyr.node  =  start ing.node.in.layer [curr ent. lay er-l]+y; 
if  (node.record [current . node] “>class  == 

node.record[preyious.lyr.node] “>class) 

nodes.of. class  +=  1; 

node.record[current_node]->connect [prcvious.lyr_node]  =  1; 

} 

else 

node.record [current .node] ->connectCprevious.lyr.node]  =  0; 
node.record [current.node] ->Be ight [previous.lyr.node]  =  0 ; 

} 

} 

for  (y  =  0;  y  <  nodes.in.layer [current .layer-1];  y++) 

{ 

previous.lyr.node  =  3tarting.node.in.layer[current.layer“l]  +  y; 
node.record  [current.node] ->ueight  [previous.lyr.node]  =  1; 

> 

} 

} 

/**^i^^*t************t*tt******r^**********t****************/ 

/*  End  Connect  Hodes  to  Class  Vodes  ♦/ 

/^^i^}^r^^0*^t^t*****************************************t*************/ 


/♦  Function  lame:  Scale.Sigmas.by.Class.Interferenco  lumber: 3. 10  ♦/ 
/*  Description:  This  function  scales  the  size  of  the  RBF  sigmas  by  */ 
/*  by  a  constant  if  a  data  point  causes  more  than  1  */ 
/*  RBF  node  to  be  excited  past  some  threshold  and  the  */ 
/*  RBF  nodes  are  not  detecting  the  same  classes.  */ 
/*  */ 
/*  Functions  Called:  7.2  calculate.layer.O. output  ♦/ 
/*  7.3  calculate.layer.l.output  ♦/ 
/*  */ 
/♦  Variables  Passed  In:  Training.Data  -  Structure  ♦/ 
/♦  Hode.Record  -  Structure  */ 
/♦  Train.Set  -  Integer  */ 
/♦  Bodes.in.Layer  -  Integer  array  */ 
/♦  Starting.lode.in.Layer  -  Integer  array  <►/ 
/*  Total.Hodes  -  Integer  ♦/ 
/♦  Current .Layer  -  Integer  ♦/ 
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/♦  Output.Threshold  “  Float  ♦/ 

/•  Scale.Factor  “  Float  ♦/ 

h  */ 

/*  Variables  Returned:  lode^Record  -  Structure  */ 

/*  Date:  10  Iot  90  Revision:  1.0  e/ 


void  scale.sigmas.by.class.interferenceCstruct  data  ^data^recorciG » 

struct  lode^data  «i.recordC3, 
int  record.no, 

int  nodes. in.layer CD » 
int  starting.node.in.layer[] , 
int  total.nodes, 
int  current.layer, 
float  out .max, 

float  scale.factor) 

{ 

int  record.ptrCTOTAL.lODES] ; 

int  X,  y,  node,  record,  z,  current .node ; 

for  (record  «  0;  record  <  record.no;  record++) 

{ 

calculato.layer.0.output(data.record, 

I.record, 
nodes. in.layer, 
record) ; 

calculate.layer.l.outputCdata.record, 

I.record, 
nodes.in.layer, 
starting.node.in.layer, 
total.nodes) ; 


X  =  0; 

for  (node  »  0;  node  <  nodes.in.layer [current. lay er] ;  node++) 

{ 

current .node  =  starting.node.in.layer [current .layer]  +  node; 
if  ((I.record [current .node] ->output  >  out .max)  kk 

(l.record[current.node]->class  !=  data.record [record] ->clas8)) 

{ 

record.ptr[x]  =  current.node; 

X  =  x+1; 

} 

} 

if  (x  >  0) 

{ 

for  (y  =  0;  y  <  x;  y++) 

{ 

current  .Xiode  =  record.ptr[y] ; 
do 
{ 

for  (z=0;  z  <  total.nodes;  z++) 

I.record [current.node] ->sigma [z]=l .record [current.node] ->sigma [z] 
-scale.factor  ♦  ( I.record [current.node] ->sigma [z] ) ; 
calculate.node.output(data.record, 

I.record, 
current.node, 
total.nodes) ; 

> 

uhilc ( H.rc c ord [curr cnt. node] -> out put  >  out .max); 

} 

} 

} 

> 

/^t^^********t***tt**  End  Optimize  Sigmas  ****************/ 
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/*  Function  lamo:  Sot^Sigma^to.Constaoit  lumber:  3.11  «/ 
/«  Description:  This  function  sets  the  RBF  sigmas  to  a  constant  ♦/ 
/«  Functions  Called:  lone  ♦/ 
/<•  Variables  Passed  In:  lodo.Record  -  Structure  ♦/ 
/♦  lodes.in^Layer  -  Integer  array  ♦/ 
/♦  Start ing^Iode.in^Layer  "  Integer  array  */ 
/♦  Current^Layer  -  Integer  ♦/ 
/*  Sigma.Constant  -  Float  ♦/ 
/*  */ 
/*  Variables  Returned:  Vode.Record  ~  Structure  ♦/ 
/♦  Date:  10  Mot  90  Revision:!. 0  ♦/ 


void  set.sigmas.to.constant (struct  lode.data  vnode.record[]  | 

int  nodes.in^layerC] , 
int  8tarting.node.in.layerD » 
int  current flayer, 
float  sigma.constant) 

{ 

int  X,  y,  current. node,  previou3.layer.node; 
for  (x=s  0;  X  <  nodes.in.layer [current .lay or] ;  x++) 

{ 

current  .node  ~  starting.node.in.layer  [current  .layer] -i-x; 
for  (y  =*  0;  y  <  nodes.in.layer[curr€nt.layer“l];  y++) 

{ 

previous.layer.node  =  st art ing.node.in.layer [current .lay er-l]+y; 
node.record[current.node]->sigma[previous.layer.node]  «  sigma.constant ; 

} 

> 

} 

Set  Sigmas  to  a  Constanteeeveeeeeeev/ 


/*  Function  laffle:Set.Sigmas.at.P.Ieighbor8.Avg  lumber :3. 12  «/ 
/*  Description:  This  function  sets  the  sigmas  of  the  RBFs  equal  to  */ 
/*  the  root  mean  square  distances  of  the  closest  P  */ 
/*  leighbors  ♦/ 
/♦  sigma  =  sqrt[(l/P)sua(dp)]  ♦/ 
/♦  ♦/ 
/♦  Functions  Called:  8.18  f ind.distance.betueen.nodes  ♦/ 
/♦  8,19  8ort.2.dim.array  ♦/ 
/♦  ♦/ 
/*  Variables  Passed  In:  lode.Record  -  Structure  «/ 
/♦  lodes.in.Layer  -  Integer  array  »/ 
/*  Starting.Iode.in.Layer  -  Integer  array  ♦/ 
/♦  Current  .Layer  -  Integer  ♦/ 
/♦  Total.lodes  -  Integer  ♦/ 
/♦  P.Ieighbors  -  Integer  ♦/ 
/♦  t/ 
/♦  Variables  Returned:  lode.Record  -  Structure  ♦/ 
/♦  Date:  10  lov  90  Revision:  1.0  ♦/ 


/iif^>^*i^*^^i^^t**************************************t*****************/ 

void  set.sigma.at.P.neighbor.avg (struct  lode.data  ♦node.recordD , 

int  nodes.in.layerD , 

int  starting.node.in.layern , 

int  current .layer, 

int  total.nodes, 

xnt  p.nc^ghbcrs] 

{ 

int  X,  y,  z,  current.node ,  next. node,  previous.layer.node; 
float  distance.between[TRAII.SET] [TRAII.SET] ; 
float  ♦distance.ptr [TRAII.SET] ; 
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double  avg>  distance ,  buffer; 
double  exponent.l  =  2; 
double  exponent. 2  =  .S; 

for  (x  =  0;  X  <  nodes.ln.layerCcurrent.layer] ;  x++) 
dist ance.pt r[x]  =  ftdistance.betveenCx] [0] ; 
for  (xa  0;  X  <  node s.in.layer [current .layer] ;  x++) 

{ 

current.node  =  st  art  ing.node.in.layer  [current  .layer]  +  x; 
for  (y  =  0;  y  <  nodes.in.layer [current. lay er] ;  y++) 

next. node  =  starting.node.in.layer[current.layer]  +  y; 
f lnd.distance.betneen.nodes (node.record , 

nodes.in.layer , 

starting.node.in.layer, 

current.node, 

next.node, 

current .layer, 

distance.ptr) ; 

} 

sort.2.dim.array (distance.ptr, 

nodes.in.layer [current .layer] , 

x); 

distance  a  O; 

for  (y  a  0;  y  <  p.neighbors  +1;  y++) 

{ 

buffer  a  distance.between[x] [y] ; 

distance  =  distance  +  pou (buffer, exponent.l) ; 

} 

avg  a  distance /p.neighbors; 
for  (2=0;  2  <  total.nodes;  2++) 
node.record [current.node] “>sigma [2]  a  pow (avg, exponent. 2) ; 

} 


/j^^t^ill^^^t*****************0*’^***t****************T¥*******/ 

/♦  End  Set  Sigma  at  P  neighbor  average  ♦/ 

/*ilf*^^i^tt*********t*****t*************i^**^t***************/ 


G.4  NETINPUT 


/*  Module  Name:  lETIRPUT.C 

Humber: 4.0 

♦/ 

/* 

Description:  This  module  provides  the  functions  necessary  to 

♦/ 

/* 

randomly  load  the  input  data  and  to  preset  the 

♦/ 

/* 

network  parameters 

♦/ 

/* 

*/ 

/* 

Modules  Called:  HETIHIT.C 

♦/ 

/♦ 

Functions  Contained:  4.1 

load.input .patterns 

*/ 

/♦ 

4.2 

load.separate.files 

*/ 

/♦ 

4.3 

load.from.single.file 

*/ 

/* 

4.4 

load.by.classes 

♦/ 

/* 

4.5 

get .data 

♦/ 

/♦ 

4.6 

normalixe.data 

♦/ 

/* 

4.7 

randomize.records 

*/ 

/* 

4.8 

get.random.class.record 

*/ 

/* 

4.9 

get.random.record 

*/ 

/* 

4.10 

calulate.euclidean.dist^tnce.betveen 

*/ 

/* 

.inputs 

*/ 

/♦ 

4.11 

get.weights 

*/ 
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/* 

4.12  get.sigmas 

♦/ 

/* 

4.13  get. outputs 

♦/ 

/* 

*/ 

/*  Dat«:  11  lov  90 

Revision:  1.0 

*/ 

tinclude  "netvrble  ,h'' 
tinclude  "netfnctn.h" 
tinclude  <tine.h> 
tinclude  <stdlib.h> 

/^*%^%:^tt**************************4*****J^*^**tt**t****tt***********/ 


/♦  Function  Hane:  load.input. pat  terns  lumber:  4.1  ♦/ 
/«  Description:  This  function  determines  wether  the  data  should  */ 
/*  be  loaded  randomly  from  separate  files ^  from  a  */ 
/*  single  file  for  by  class  */ 
/♦  */ 
/♦  Functions  Called:  4.2  load^separate.files  */ 
/♦  4.3  load.from^single.file  ♦/ 
/♦  4.4  load.by. classes  ♦/ 
/♦  ♦/ 
/♦  Variables  Passed  In:  training.data  -  Structure  array  ♦/ 
/*  test.data  -  Structure  array  ♦/ 
/*  train^set  -  Integer  ♦/ 
/♦  test^set  -  Integer  */ 
/♦  dimension  -  Integer  */ 
/*  classes  -  Integer  */ 
/♦  training.patterns^in. class  -  Integer  array  ♦/ 
/♦  randomization.rule  -  Integer  ♦/ 
/♦  data.seed  -  Unsigned  ♦/ 
/♦  ♦train.ptr  -  FILE  pointer  ♦/ 
/*  ♦test.ptr  -  FILE  pointer  ♦/ 
/♦  */ 
/♦  Variables  Returned:  *train^ptr  ~  FILE  pointer  ♦/ 
/♦  *test^ptr  ~  FILE  pointer  ♦/ 
/♦  Date:  11  Nov  90  Revision:  1.0  */ 


void  load.input.patternsC struct  data  *training.data[] , 
struct  data  *test^data[] , 
int  train.set, 
int  test.set, 
int  dimension, 
int  classes, 

int  training^pattems.in^classCl , 
int  randomization.rule, 
unsigned  data.seed, 

FILE  *train.ptr, 

FILE  ♦test.ptr) 


{ 

int  random.record[TRAIH.SET+TEST_SET] ; 

if  (random '.zation.rule  ==  1) 

{ 

load.separate.files  (training.data, 
test.data, 
tmin.sct , 
test.set , 
dimension, 
randoffl.record, 
data.seed, 
train.ptr, 
test.ptr) ; 
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} 

else  if  (randomization^rule  ==  2) 

load_froin.single.file  (training.data, 
test.data, 
train.set, 
test.set , 
dimension, 
randooi.record, 
data.seed, 
train.ptr) ; 

> 

else  if  (randomization.rule  ==  3) 

{ 

load.by.classes  (training.data, 
test.data, 

training.patterns.in.class , 

random.record, 

train.set , 

test.set, 

dimension, 

classes, 

data.seed, 

train.ptr) ; 

} 

else 

{ 

printf (”\nerror  in  randomization  rule"); 

} 

> 


/ii^i^t********************tt*****************f^*i^t****^’¥**t***/ 

/*  End  Load  Input  patterns  */ 

/♦  Functions  called  by  Load  Input  patterns  ♦/ 

/f^^*t*******************^*****************nf***********************t>/ 

/*  Function  Hame:  load.separate.files  Humber:  4.2  */ 

/*  Description:  This  function  loads  the  input  data  from  a  separate  */ 
/*  test  and  training  file  randomly.  */ 

/*  */ 

/♦  Functions  Called:  4.5  get.data  ♦/ 

/♦  4.7  randomize.records  ♦/ 

/♦  ♦/ 

/♦  Variables  Passed  In:  training.data  -  Structure  array  ♦/ 

/*  test.data  -  Structure  array  ♦/ 

/*  train.set  -  Integer  ♦/ 

/♦  test.set  -  Integer  */ 

/♦  dimension  -  Integer  ♦/ 

/♦  random.record  -  Integer  array  */ 

/♦  data.seed  -  Unsigned  ♦/ 

/♦  *train.ptr  -  FILE  pointer  ♦/ 

/♦  ♦test.ptr  -  FILE  pointer  */ 

/♦  ♦/ 

/♦  Variables  Returned:  training.data  -  Structure  array  ♦/ 

/♦  test.data  -  Structure  array  ♦/ 

/*  Date:  11  Bov  90  Revision:  1.0  ♦/ 
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void  load.soparate.f lies  (struct  data  *training.dataC3 , 
struct  data  ♦test^data[] , 
int  train^set, 
int  test. set, 
int  dimension, 
int  random.recordC] , 
unsigned  seed, 

FILE  ♦train.ptr, 

FILE  ♦test.ptr) 

{ 

randomize.re cords (train.set , 

random.r ecord , 
seed) ; 

get.dataCtraining.data, 
train.set, 
dimension, 
train.ptr, 
random.record) ; 


randomize.records(test.set , 

random.record, 
seed) ; 

get. dataCtest. data, 
test.set , 
dimension, 
test.ptr, 
random.record) ; 

} 

/***t***^*******4***********4t***ti^***t*************4*****t***i^*****/ 

/*  Function  Same:  load.from.single.filo  Humber :4. 3  ♦/ 

/«  Description:  This  function  loads  the  training  and  test  data  ♦/ 

/*  randomly  from  a  single  file  ♦/ 

/*  ♦/ 

/♦  Functions  Called:  5.7  create.data.record  ♦/ 

/*  4.5  get.data  ♦/ 

/♦  4.7  randomize.records  ♦/ 

/*  ♦/ 

/*  Variables  Passed  In:  training.data  -  Structure  array  ♦/ 

/♦  test.dat a  -  Structure  array  ♦/ 

/♦  train.set  -  Integer  ♦/ 

/♦  test.set  ~  Integer  »/ 

/♦  dimension  -  Integer  */ 

/♦  random.record  -  Integer  array  */ 

/*  data.seed  -  Unsigned  */ 

/♦  ♦train.ptr  -  FILE  pointer  ♦/ 

/♦  ♦test.ptr  “  FILE  pointer  ♦/ 

/*  ♦/ 

/♦  Variables  Returned:  training.data  ~  Structure  array  ♦/ 

/♦  test.data  -  Structure  array  ♦/ 

/♦  */ 

/*  Date:  11  Bov  90  Revision:  1.0  */ 

void  load.from.single.filo  (struct  data  ♦training.data [] , 

struct  data  ♦test.dataC] , 

int  train.set, 

int  test.set, 

int  dimension, 

int  random.recordC], 
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misigned  seed» 
FILE  ♦train^ptr) 


{ 

int  X,  y; 

int  error  »  0; 

struct  data  ♦teinp«dataCTEST„SET+TRAII«SET] ; 

for  (x  =  0;  X  <  traiii_set+test«sGt ;  x++) 

{ 

Croat e.data.recordCtemp.data, 
terror) ; 

if  (error  !=  0) 

{ 

printfC'Xn  ♦♦  Out  of  memory  for  temp  data  ♦♦  \n'*); 

} 

} 

randomize.recordsCtrain.set  +  test.set, 
random.record , 
seed) ; 

get.data  (temp.data, 

test«set  +  train^set, 
dimension, 
train^ptr, 
r and om.re cord) ; 

for  (x  =  0;  X  <  train.set;  x++) 
training.dataCx]  =  temp.dataW; 

for  (x  =5  0;  X  <  test. set;  x++) 

y  =  train.set  +  x; 
test.dataCx]  =  temp.dataCy] ; 

} 

} 

/^^^Tlit***************t**J¥*********************tHH^^tt**J$*****t********/ 

/*  Function  lame:  load.by.classes  lumber:  4.4  */ 

/*  Description:  This  function  loads  a  user  selected  number  of  */ 

/♦  training  patterns  for  each  class  raoidomly  from  a  ♦/ 

/♦  single  file.  The  remaining  patterns  are  loaded  ♦/ 

/♦  as  test  patterns  ♦/ 

/♦  ♦/ 

/♦  Functions  Called:  5.7  create.data.record  ♦/ 

/♦  4.5  get.data  ♦/ 

/♦  4.7  randomize.records  ♦/ 

/♦  4.9  get_random.record  ♦/ 

/♦  ♦/ 

/♦  Variables  Passed  In:  training.data  -  Structure  array  m/ 

/♦  test.data  -  Structure  array  ♦/ 

/♦  training.patterns.in.class  -  Integer  array  ♦/ 

/♦  random.record  -  Integer  array  ♦/ 

/♦  train.set  -  Integer  */ 

/♦  test.set  -  Integer  ♦/ 

/♦  dimension  -  Integer  ♦/ 

/♦  classes  -  Integer  ♦/ 

/♦  data.seed  -  Unsigned  ♦/ 

/*  ♦train.ptr  -  FILE  pointer  ♦/ 

/♦  */ 

/*  Variables  Returned:  training.data  -  Structure  array  */ 

/♦  test.data  -  Structure  array  */ 

/♦  */ 
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/*  Date:  11  lov  90  Revision:  1.0  ♦/ 

void  load^by^classes (struct  data  ♦training.dataC] , 
struct  data  ^test.dataQ, 
int  training.patterns.in.class[] , 
int  random^recordC] , 
int  train.set, 
int  test.set, 
int  dimension, 
int  classes, 
unsigned  seed, 

FILE  *fptr) 

{ 

struct  data  *tefflp«dataCTRAII^SET+TEST^SET] ; 

int  number^in.class [CLASSES] ; 

int  X,  y,  record; 

int  error  =  0; 

int  class  -  0; 

srand(seed) ; 

for  (x  3  0;  X  <  classes-M;  x++) 
number.in.classCx]  =  0; 

for  (x  =  0;  X  <  train.set  +  test. set;  x++) 

{ 

create.data.recordCtemp.data, 

X, 

terror) ; 

if  (error  !=  0) 

{ 

printf("\n  ♦♦  Out  of  memory  for  temp  data  ♦♦  \n”); 

} 

} 

randomize. re cords (train.set+test.set , 
random.record, 
seed); 

get.data(temp.data, 

train.set+test.set , 

dimension, 

fptr, 

random.record) ; 

X  =  0; 
do 
{ 

get.random.record(classes , 

Aclass) ; 


class  =  class  +  1; 

number.in.class [class]  =  number.in.class [class] +1; 
if  (training.patterns.in.class [class] +1  >  nximber.in.class [class]) 
{ 

do 

get.randora.record(train.set+test.sct, 

Arecord) ; 

while (t emp.dat a [record] ->class  !=  clans); 

training.dat a [y]  =  t emp.dat a  [record] ; 

temp.dat a [record]  =  temp.dat a [train.set  +  test.set  -1]; 

train.set  =  train.set  -1; 

X  =  X  +  1; 

} 

else 

number.in.class [class]  =  training.patterns.in.class [class] ; 
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} 

while  (train.set  >  0) ; 


for  (x  =  0;  X  <  test.set;  x++) 
test.data[x3  **  temp.dat a [x] ; 


/*  End  Functions  Called  by  Load  Input  Patterns  ♦/ 

/*m******^*0ti***********t*****‘*t*************4****0»*ilr*^***/ 

/^^^*t*t*t**^****^t********t>^*^4*t0**t*********t***fj***nf************/ 

/*  Function  lame:  get. data  Humber:  4.5  */ 

/*  Description:  This  function  loads  the  data  into  an  array  given  */ 
/*  by  the  user.  ♦/ 

/*  */ 

/e  Functions  Called:  lone  ♦/ 

/♦  Variables  Passed  In;  t raining.dat a »  test.data  -  Structure  array  ♦/ 
/♦  traiu.set  or  test.set  -  Integer  «/ 

/♦  dimension  -  Integer  ♦/ 

/♦  ♦train.ptr  or  ♦test.ptr  ~  FILE  pointer  ♦/ 

/♦  random.record  -  Integer  array  */ 

/♦  ♦/ 

/♦  Variables  Returned:  training.data^  test.data  -  Structure  array  ♦/ 

/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


/^^i$0^^^^t***********t*^**********^i^****i^******i^*******ty^***********/ 

void  get.data  (struct  data  ♦data.recordC3> 
int  record.no I 
int  dimension, 

FILE  *fptr, 

int  random.recordG  ) 


{ 

int  X,  y,  record; 

float  vector.data; 

int  Xnovn.class; 

for  (x  =  0;  X  <  record.no;  x++) 

{ 

record  =  random.record[x3 ; 
for  (y  =  0;  y  <  dimension;  y++) 

{ 

fscanf(fptr,  “%f",  Jtvector.dat a) ; 
data_recordCrecord3“>vectorCy3  =  vector.data; 

} 

fscanfCfptr,  "Xd'*>Hknown.class) ; 
data.record[record3“>clas8  =  known.class; 
data.record[record3“>number  =  x; 

> 

} 

/*****0t****t**<^***i**t*^^***t*******************************ttinii***0**/ 


/*  Function  lame:  normalize.dat a  lumber:  4.6  */ 
/«  Description:  This  fuikction  energy  normalizes  each  component  of  */ 
/♦  the  input  data  by  ♦/ 
/♦  x(k)  =  sqrt{Cx(k)-23/Ilx!l-2>  ♦/ 
/♦  Functions  Called:  Hone  ♦/ 
/♦  */ 
/»  Variables  Passed  In:  training.data,  test.data  -  Structure  array  ♦/ 
/♦  train.set  or  test.set  -  Integer  ♦/ 
/♦  dimension  -  Integer  •/ 
/♦  */ 
/*  Variables  Returned:  training.dat a,  test.data  -  Structure  array  */ 
/*  Date:  11  lov  90  Revision:  1.0  */ 
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void  normalize.dat &  (struct  data  *data.rocordG > 
int  record.no, 
int  di&ension) 


double  buffer; 
float  distance; 
double  exponent. 2  »  2; 
double  exponent. 1  =  .5; 
int  X,  y; 

for  (x  =  0;  X  <  record.no;  x++) 
buffer  =  0; 

for  (y  =  0;  y  <  dinension;  y++) 

buffer  =  buffer  +  po«((double)  (dat a.rec or dCx]->vect or [y] ), exponent .2) ; 
distance  »  pov(buffer«  exponent.l); 

for  (y  =  0;  y  <  dimension;  y++) 

data_recordEx]~>vector[y]  =  d at a.r c c ord Cx]->vect or [y] /distance; 
buffer  =  0; 

for  (  y  =  0;  y  <  dimension;  y++) 

buffer  =  buffer  +  pou((double}  (data.record [x]“> vector [y] ), exponent. 2) ; 
distance  =  pou (buffer, exponent.l) ; 

> 

} 


/*  Function  Same:  randcmize.records  Sitnber:  4.7  */ 

/♦  Description:  This  function  returns  an  array  containing  random  ♦/ 

/*  from  0  to  the  number  of  input  patterns  numbers  ♦/ 

/♦  ♦/ 

/♦  Functions  Called:  4.9  got.random_record  ♦/ 

/♦  ♦/ 

/♦  Variables  Passed  In;  train.set  or  test.set  -  Integer  ♦/ 

/*  random.record  -  Integer  array  ♦/ 

/♦  data.seed  -  Unsigned  ♦/ 

/♦  */ 

/♦  Variables  Returned:  random.record  -  Integer  array  */ 

/♦  Date:  11  Hov  90  Revision:  1.0  ♦/ 


void  randomize.records(int  max.no, 

int  raoidom.recordD , 
unsigned  seed) 

{ 

int  X,  record,  tempCTRAIS.SET+TEST.SET] ; 
for  (x  =  0;  X  <  max.no;  x++) 

random.record [x]  =  x; 
tempCx]  =  x; 

} 

X  =  0; 

if  (seed  !=  0) 

{ 

srand(seed) ; 
do 
{ 

get_random.record(max_no , 

^record) ; 

random.record [x]  =  temp [rr cord]; 
tempErecord]  =  teiDpEniax.m  -i]  ; 
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inax.no  *  nax.no-l; 
X  =  X  +  1; 

} 

while  (nax.no  >  0); 

> 

} 


/e  Function  lane:  get.randoffl.clas8.record  lumber:  4.8  */ 
/*  Description:  This  function  returns  the  random  class  and  record  */ 
/♦  number  for  a  pattern  with  that  class  */ 
/♦  */ 
/♦  Functions  Called:  4.9  get.randosurecord  ♦/ 
/*  Variables  Passed  In:  training  or  test.data  -  Structure  array  */ 
/♦  train.set  or  test.set  -  Integer  ♦/ 
/*  classes  -  Integer  */ 
/*  ^record  -  Integer  pointer  ♦/ 
/♦  */ 
/♦  Variables  Returned:  erecord  -  Integer  pointer  ♦/ 
/♦  Date:  11  Iot  90  Revision:  1.0  ♦/ 


void  g6t.random.class.record(struct  data  ♦data.recordC] , 

int  max.recordj 
int  max.classes, 
int  ^record) 

{ 

int  class; 
int  record.number; 
get.random.record(max.class6s, 
ftclass) ; 

class  =  class  +  1; 
do 

get.random.record(max.record, 

ftrecord.number) ; 

while (data.record [record.number] ~>class  !=  class); 
♦record  =  record.number; 

> 


/♦  Function  lame:  get_random.record  lumber:  4.9  ♦/ 
/♦  Description:  This  function  returns  a  random  number  between  0  ♦/ 
/♦  and  maximum  number  -1  ♦/ 
/♦  ♦/ 
/♦  Functions  Called:  Hone  ♦/ 
/♦  Variables  Passed  In:  max.number  -  Integer  ♦/ 
/♦  ♦record  -  Integer  pointer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  ♦record  -  Integer  pointer  ♦/ 
/♦  Date:  11  Hov  90  Revision:  1.0  ♦/ 


void  get.random.record(int  max.number, 
int  ♦record) 


float  x; 
int  buffer; 

X  =  ((float)rand())/(32767^  65534); 
buffer  =  (x  ♦  max.number  +  .5); 
if  (buffer  >  max.number  -.5) 

♦record  =0; 
else  if  (buffer  <  0) 

♦record  =  0; 
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else 

♦record  =  buffer; 

> 

/♦  Function  Va&e:  calculate.euclidean.distance.betveen  lumber: 4. iOe/ 


/♦  .inputs  ♦/ 
/♦  Description:  This  function  calculates  the  distance  between  ♦/ 
/♦  each  of  the  input  data  records  by  */ 
/♦  d(ij)  a  8qrt{sua[x(i)“  x(j)]*'2>  ♦/ 
/♦  ♦/ 
/♦  Functions  Called:  lone  ♦/ 
/*  Variables  Passed  In:  training  or  tost.data  -  Structure  array  ♦/ 
/♦  train.set  or  test.set  -  Integer  ♦/ 
/*  dimension  -  Integer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  lone  ♦/ 
/♦  Date:  11  Hoy  90  Revision:  1.0  ♦/ 


void  calculate.euclidean.distance.between.input8(struct  data  ♦data.recordG , 

int  record.no > 
int  dimension, 

FILE  ♦fptr) 

{ 

float  distance; 
double  buffer  =  0; 
double  exponent.l  =  .5; 
double  exponent. 2  a  2; 
int  X,  y,  2; 

for  (x  a  0;  X  <  record.no;  x++) 
for  (y  a  0;  y  <  record_no;  y++) 

{ 

buffer  a  0; 

for  (2  a  0;  z  <  dimension;  z++) 

buffer  a  buffer  +  pow ( (double) (data.recordCx]“>vector [2]  - 

data.recordCy]  *>vect<^r  [2]  ) ,  exponent. 2) ; 
distance  =  po w (buffer, exponent .1 ) ; 
fprintf (fptr,  “\n  y.d  y.d  distance  =  y,f  classes  a  ^d  y,d  ”, 
x,y , distance ,data.re cord Cx]“> class , 
data.recordCy] ->class) ; 

} 


/♦  Function  Same:  get.veights  lumber:  4.11  ♦/ 
/♦  Description:  This  function  reads  the  initial  network  weights  ♦/ 
/♦  from  a  file  ♦/ 
/♦  */ 
/♦  Functions  Called: Hone  ♦/ 
/♦  Variables  Passed  In:Hode.record  -  Structure  array  ♦/ 
/♦  ♦fptr  -  FILE  pointer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  lode.record  ~  Structure  array  ♦/ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


void  get.weights  (struct  Hode.data  ♦node.recordC] , 
FILE  ♦fptr) 


int  X,  y; 
float  weight; 

while (fscanf (fptr, “y,d  y*d  y,f”,  tx,  ky,  kweight)  !=  EOF) 
node.record[x]->weight [y]  =  weight; 


G-51 


> 


/♦  Function  lame;  get.sigm&s  lunber:  4.12  */ 

/#  Description:  This  function  reads  the  initial  network  sigmas  e/ 
/♦  from  a  file  */ 

/•  */ 

/♦  Functions  Called: lone  ♦/ 

/e  Variables  Passed  In:Rode^record  -  Structure  array  */ 

/*  efptr  -  FILE  pointer  ♦/ 

/♦  V 

/♦  Variables  Returned:  lode.record  -  Structure  array  ♦/ 

/♦  Date:  11  Hov  90  Rewision:  1.0  */ 


▼oid  get. sigmas (struct  lode.data  enode.recordC] , 

FILE  efptr) 

{ 

int  X,  y; 
float  sigma; 

while (f8canf(fptr»  *'Xd  Xf",  tx,  Jky,  rtsigma)  !«  EOF) 
node.recordCx3->sigma[y]  =  sigma; 

} 


A  Function  lame:  get_out;}Ut'.$  lumber:  4.13  e/ 

/«  Description:  This  function  reads  the  initial  network  outputs  v/ 
/♦  from  a  file  ♦/ 

/♦  ♦/ 

/♦  Functions  Called: lone  ♦/ 

/♦  Variables  Passed  In:lode.record  Structui*'*  array  *i 

/*  efptr  -  FILE  pointer  ♦/ 

A  ♦/ 

/♦  Variables  Returned:  lode.record  -  Structure  array  ♦/ 

A  Date:  11  lov  90  Revision:  1.0  ♦/ 


void  g6t.outputs(struct  lode.data  enode.recordC] , 
FILE  *fptr) 


{ 

int  x; 

float  output; 

while (fscanfCfptr,  "ytd  yf",l:x,  Aoutput)  !=  EOF) 
node.recordCx]->output  =  output; 

} 


G.5  NETINIT 


A  Module  lame:  lETIlIT.C  lumber :5.0  ♦/ 
/♦  Description:  This  module  provides  the  initialization  routines  ♦/ 
/*  for  the  nodes  of  a  neural  network  ♦/ 
/*  Modules  Called:  lone  */ 
/*  */ 
/*  Functions  Contained:  5.1  initialize.node.weights  «/ 
A  5.2  initialize.node.connections  ♦/ 
A  5.3  initialize.node.sigmas  ♦/ 
/♦  5.4  initialize.node.outputs  ♦/ 
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/♦  5.5  initialize.noQ.transfer.f unction  ♦/ 
/♦  5.6  create.node  */ 
/*  5.7  croate.data.record  ♦/ 
/*  5.8  disconnact.noda  ♦/ 
/♦  5.9  correct. node.veights  */ 
/♦  */ 
/•  Date;  10  lot  90  Revision:  1.0  ♦/ 


tinclude  ’’netvrble  .h" 
tinclude  "netfnctn.h" 

/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee/ 


/e  Function  lane:  initialize.node. weights  Kunber:  5.1  e/ 
/e  Description:  This  function  initializes  the  weights  between  */ 
/*  connected  nodes  to  the  range  ~1  to  1 .  For  un-  */ 
/e  connected  nodes »  the  weights  are  set  to  0  e/ 
/♦  ♦/ 
/♦  Functions  Called:  lone  ♦/ 
/e  Variables  Passed  In:  Vode.record  *  Structure  array  e/ 
/*  total.nodes  -  Integer  ♦/ 
/♦  weight.seed  -  Unsigned  */ 
/♦  ♦/ 
/♦  Variables  Returned:  lode.record  -  Structure  array  ♦/ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


void  initialize.node.weights  (struct  lode.data  enode.recordQ , 

int  totai.nodes> 
unsigned  seed) 


{ 

int  X,  y; 
float  fanout  =  0; 
double  z  =  -1.0; 
double  w  «  1.0; 

srand(seed); 

for  (x  =*  0;  X  <  total.nodes;  x++) 

{ 

fanout  =  0; 

for  (y  =  0;  y  <  total.nodes;  y++) 
if  (node.recordCx]->connectCy3  ==  1) 
fanout  =  fanout  +  1; 
for  (y  =  0;  y  <  total.nodes;  y++) 
if  (xiode.recordCx]->connoct[y]  ==  1) 

{ 

w  =  (double) rand( ) ; 

node.record[x]->weightCy3  =  (pow (z,w)*( (float )rand()) / (65534*32767) ) ; 

} 

else 

node.record[x3->weight[y]  =  0; 

} 


/*t^^^t^t******0********f^i^**0**i^ti**t*********t**********^***r^*********/ 

/*  Function  Vane:  initialize_node_conn6Ctions  Vunber:  5.2  */ 
/*  Description:  This  function  initalizes  the  connections  between  */ 
/*  nodes  which  should  be  connected  to  1.  The  nodes  */ 
/♦  which  shouldn't  be  connected  are  set  to  0.  These  ♦/ 


/*  connections  are  dependent  on  network  topology  */ 
/*  */ 
/*  Functions  Called:  lone  */ 
/♦  Variables  Passed  In:  Vode.record  -  Structure  array  ♦/ 
/♦  number.of. layers  -  Integer  ♦/ 
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/♦  nodes. in.layer  -  Integer  array  ♦/ 
/♦  8tarting.node.in.layer  -  Integer  array  ♦/ 
/*  network.type  Integer  ♦/ 
/♦  total.nodes  -  Integer  */ 
/*  */ 
/♦  Variables  Returned:  lode.record  -  Structure  array  */ 
/•  Date:  11  lov  90  Revision:  1.0  ♦/ 


void  initialize.node.connection8(htruct  lode.data  vnode.recordQ » 

int  number.of .layers , 
int  nodes. in.layerQ , 
int  starting.node.layerC] , 
int  netvork.type, 
int  total.nodes) 


int  layer.no,  x,  y,  layer; 

switch  (network.type) 

{ 

case  1: 

for  (x  «  0;  X  <  total.nodes;  x++) 
for  (  y  «  0;  y  <  total.nodes;  y++) 
node.record[x]~>connectCy3  *  0; 

for  (layer.no  =  1;  layer.no  <  nuaber.of. layers  +1;  layer.no++) 

for  (x  «  st art ing.node.layer [layer .no] ;  x  <  8tarting.node.laytr[layer.no]+nodes.in.layer[layer.no]  ;  x++) 
for  (layer  »  0;  layer  <  nunber.of. layers  +  1;  layer  ++) 

for  (  y  =  start ing.node.layer [layer] ;  y  <  start ing.node.layer[layer]'fnode8.iA.layer [layer] ;  y++) 
if  (layer  ==  layer.no  -1) 
nodo.record[x]->connect[y]  *  1; 
else 

node.record[x]“>connect[y]  «  0; 

break; 

default: 

printf("\n  Error  in  network  selection"); 
break; 

} 

} 


/e  Function  lamo:  initialize.node.sigmas  lumber:  5,3  */ 
/*  Description:  This  function  initializes  the  node  sigmas  to  a  *t 
/e  random  value  between  0  and  1  */ 
/♦  ♦/ 
/♦  Functions  Called:  lone  ♦/ 
/♦  Variables  Passed  In:  lode.record  -  Structure  array  ♦/ 
/♦  total.nodes  -  Integer  ♦/ 
/♦  sigma.seed  -  Unsigned  ♦/ 
/♦  */ 
/♦  Variables  Returned:  lode.record  -  Structure  array  ♦/ 
/♦  Date:  11  Hov  90  Revision:  1.0  ♦/ 


void  initialize.node.sigmas (struct  Vode.data  ♦node.recordQ , 

int  total.nodes, 
unsigned  seed) 


int  X,  y; 
srand(seed) ; 

for  (x  =  0;  X  <  total.nodes;  x++) 
for  (y  =  0;  y  <  total.nodes;  y++) 
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xiod€^record[x3“>sigaaCy]  »  (( (float) randO ) /(32767^65534) ) ; 

> 


Function  Iam«;  initialize.node.outputs  lunber:  5.4  >»/ 
/*  Description:  This  function  initializes  the  outputs  for  all  the  e/ 
/♦  nodes  to  0,  */ 
/♦  */ 
/♦  Functions  Called:  lone  */ 
/♦  Variables  Passed  In:  lode^record  -  Structure  array  */ 
/*  total.nodes  -  Integer  ♦/ 
/•  ♦/ 
/♦  Variables  Returned:  lode.record  -  Structure  array  ♦/ 
/♦  Date:  11  Iot  90  Reyision:  1.0  ♦/ 


void  initialize^node^outputs (struct  lode.data  enode.xecordC] > 

int  total.nodes) 

{ 

int  node; 

for  (node  ^  0;  node  <  total.nodes;  node+'t-) 
node«record[node3->output  =  0; 

} 

/♦  Function  lame:  initialize«node»transfer.f unction  lumber:  5.5  ♦/ 


/♦  Description:  This  function -initializes  the  transfer  fxmction  ♦/ 
/*  for  each  node  in  the  network.  These  transfer  ♦/ 
/e  functions  are  depenedent  on  the  layer  the  node  */ 
/♦  is  assigned.  */ 
/♦  ♦/ 
/♦  Functions  Called:  lone  ♦/ 
/♦  Variables  Passed  In:  Hode.record  -  Structure  array  ♦/ 
/♦  number_of flayers  -  Integer  ♦/ 
/♦  nodes.in.layer  -  Integer  array  ♦/ 
/♦  starting.node^in.layer  -  Integer  array  »/ 
/♦  transfer.function  ~  Integer  array  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  lode.record  -  Structure  array  ♦/ 
/♦  Date:  11  lov  90  Reyision:  1.0  ♦/ 


void  initialize^node.transfer.funct ion (struct  lode.data  «node.record[3 , 

int  number.of. layers, 
int  nodes«in«layerC3 , 
int  starting.node.layerD , 
int  transfer_functionC3) 


{ 

int  layer,  node; 

for  (layer  =  0;  layer  <  number.of. layers  +1;  layer++) 
for  (node  =  starting.node»layerClaycr3 ;  node  <  starting.node_layer[layer3+nodes^in_layerClayer3 ;  node++) 
nodc^recordEnode3“>transfer^f unction  =  transf er^functionClayer3 ; 

} 


/♦  Function  lame:  create.node  Humber:  5.6  */ 
/e  Description:  This  function  creates  a  data  structure  for  each  «/ 
/*  node  in  the  iietwork  */ 

h  ♦/ 

/♦  Functions  Called:  Hone  ♦/ 
/♦  Variables  Passed  In:  lode.record  -  Structure  array  ♦/ 
/♦  new.node  -  Integer  */ 
/♦  ♦error  -  Integer  pointer  ♦/ 


/♦  ♦/ 
/♦  Viiriables  Returned:  ♦error  -  Integer  pointer  ♦/ 

/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 

void  create^nodeC struct  lode^data  ♦node^recordf] , 
int  new.node, 
int  ♦error) 


{ 

if  ((node.recordCneu.node]  =  (struct  lode.data  ♦)malloc(sizeof (struct  lode.data)))  ==  HULL) 
♦error  «  1; 

} 


/♦  Function  lame:  create.dat a.record  lumber:  5.7  ♦/ 
/♦  Description:  This  function  creates  a  data  structure  for  each  ♦/ 
/♦  training  and  test  pattern  used  in  the  network  ♦/ 
/♦  ♦/ 
/♦  Functions  Called:  Hone  ♦/ 
/♦  Variables  Passed  In:  lode.record  ~  Structure  array  ♦/ 
/♦  neu.record  -  Integer  ♦/ 
/♦  ♦error  -  Integer  pointer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  ♦error  -  Integer  pointer  ♦/ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


void  create.data.record (struct  data  ♦dat a.record  □  , 
int  neu.record) 
int  terror) 

{ 

if ((dat a.record [neu.record] =  (struct  data  ♦)malloc(sizeof (struct  data))) 

==IULL) 


} 


♦error  =  1; 


/♦  Function  lame:  disconnect  .node  Humber:  5.8  ♦/ 
/♦  Description:  This  function  disconnects  any  nodes  which  are  ♦/ 
/♦  no  longer  required  by  the  network  ♦/ 
/♦  ♦/ 
/♦  Functions  Called;  Hone  ♦/ 
/♦  Variables  Passed  In:  Hode.record  ~  Structure  array  ♦/ 
/♦  current .node  -  Integer  ♦/ 
/♦  total.nodes  -  Integer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  Hode.record  -  Structure  array  ♦/ 
/♦  Date:  11  Hov  90  Revision:  1.0  ♦/ 


/**^***^*^**^*^i^**^i^iH^Hit**t********************00*****ttt***********/ 

void  disconnect. node (struct  Hode.data  ♦node.recordC] , 
int  current .node, 
int  total.nodes) 

{ 

int  x; 

for  (x  =  0;  X  <  total.nodes;  x++) 

{ 

node_recordCx]->connect [current .node]  =  0; 
node.record[currcnt.node]->connect [x]  =  0; 

} 

} 


/t******t************f^***************^******************************/ 

/♦  Function  Hame:  correct. node.weights  Humber:  5-9  ♦/ 
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/*  Description:  This  function  ensures  the  weights  always  range  ♦/ 
/♦  between  -100  and  100  ♦/ 
/♦  ♦/ 
/«  Functions  Called:  lone  «/ 
/*  Variables  Passed  In:  lode.record  -  Structure  array  ♦/ 
/♦  total.nodes  -  Integer  ♦/ 
/♦  ♦/ 
/*  Variables  Returned;  Xode.record  -  Structure  array  ♦/ 
/*  Date:  11  Hov  90  Revision:  1.0  v/ 


void  correct^node^weights (struct  lode.data  ♦node«rccord[3  , 
int  total_nodes) 


{ 


int  X,  y; 

for  (x  =  0;  X  <  total.nodes;  x++) 
for  (y  =  0;  y  <  total.nodes;  y++) 
if  (node.re cord Cx]-> weight  [y]  >  0) 

{ 

if (node.record[x] -> weight Cy]  >  100) 
node.rccord[x]->weightCy3  =  100; 
else  if  (node.record[x]->weightCy]  <  .0001) 
node.recordCx3->weightCy]  =  .0001; 

} 

else 

{ 

if  (node_rccord[x3->weightCy3  <  -100) 
node.record Cx3“>w eight Cy3  =  -100; 
else  if  (node.record [x3“>weightCy3  >  -.0001) 
node.record [x3-> weight [y3  =  -.0001; 

} 


G,6  NETSHOW 


/* 

Module  Same:  BETSHOV.C  lumber:  6.0 

*/ 

/♦  Description:  This  module  contains  the  functions  which  display 

♦/ 

/♦ 

or  file  network  data. 

♦/ 

/* 

♦/ 

/v  Modules  Galled:  lone 

»/ 

/* 

Functions  Contained: 

h 

6.1 

file.data 

6.2  file.randomization.rul6 

♦/ 

/* 

6.3 

file.seeds 

6.4  file.net. topology 

♦/ 

/♦ 

6.5 

f ile. t ransf er.f unct ions 

6.6  f ile.nodcs.at.data.points.info+Z 

/♦ 

6.7 

file.center.at.avgs.dat a  6.8  f ile.k.me^s.data 

*/ 

/* 

6.9 

file.kohonen.dat a 

6.10  file.MSE.data 

♦/ 

/* 

6.11 

file.CFM.data 

6.12  file.CE.dat a 

/* 

6.13 

f ile .mat r ix.dat a 

6.14  file.parzen.window.dat a 

*/ 

/* 

6.15 

file.sigma.data 

6.16  print .last.layer.output 

♦/ 

/* 

6.17 

f3le.last.layer.output 

6.18  print.dat a 

*/ 

h 

6.19 

file.data 

6.20  print .node.dat a 

*/ 

h 

6.21 

filc.nodc.dat a 

6.22  prxnt .nodc.wcxghts 

«/ 

/* 

6.23 

file.node.weights 

6.24  print .node. sigma 

*/ 

/♦ 

6.25 

print .node.output 

6.26  pr int .node.t ransf er.f unct ion 

*/ 

/♦ 

6.27 

file.network.paramoters 

6.28  f ile.error.data 

♦/ 

/* 

6.29 

file.class.count 

*/ 

/♦ 

♦/ 

/* 

Date: 

11  Hov  90  Revision: 

1.0 

♦/ 
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tinclude  "netvrble .h“ 
tinclude  "netfnctn.h** 

/}^^^^^^0^^^t*4**********************tt*^^***t***********t**********i$/ 

/♦  Function  lame;  fi3e.data.param6ters  lumber;  6.1  ♦/ 

/♦  Description:  This  function  files  the  data  parameters  such  as  ♦/ 

/«  name  of  data  files >  lengthy  dimension  and  classes.  */ 


/* 

♦/ 

/♦  Functions  Called;  lone 

♦/ 

/♦  Variables  Passed  In: 

train^file  -  Character  array 

*/ 

/* 

test.file  “  Character  array 

♦/ 

/* 

train.set  -  Integer 

*/ 

/* 

test.set  ~  Integer 

♦/ 

/♦ 

dimension  -  Integer 

♦/ 

/♦ 

classes  Integer 

♦/ 

/* 

♦fptr  -  File  pointer 

♦/ 

/♦  Variables  Returned: 

*/ 

/♦  Date:  11  lov  90 

Revision:  1.0 

♦/ 

void  file«data^parameters(char  train.fileC] , 
char  test«fileD, 
int  train.set, 
int  test.set, 
int  dimension, 
int  classes, 

FILE  ♦fptr) 

{ 

fprintf (fptr,"\n  Training  file  =  Xs”,train^file) ; 
fprintf (fptr,“  Test  file  =  ^^s”,  test.file); 
fprintf  (fptr,  “\n  with  %d  training  vectors  and  y.d  test  vectors'*, 
train.8et,test.set) ; 

fprintf  (fptr,  "\n  dimension  =  y,d  classes  =  y,d'*, 
dimension,  classes); 
fprintf  (fptr ,  '*\n") ; 

} 

/0f^t*^^^t*************t******************************************^t***/ 


/*  Function  lame:  file.randomization.rule  lumber:  6.2  */ 
/*  Description:  This  function  files  the  method  by  which  the  data  */ 
/♦  was  loaded.  ♦/ 
/♦  ♦/ 
/*  Functions  Called:  lone  ♦/ 
/*  Variables  Passed  In:  randomization^rule  ~  Integer  */ 
/*  training_patterns_in«class  -  Integer  array  ♦/ 
/*  classes  -  Integer  ♦/ 
/♦  ♦fptr  -  File  pointer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  lone  ♦/ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


void  file^randomization.rule(int  randomization^rule, 

int  training«pattcrns^in^classQ , 
int  classes, 

FILE  ♦fptr) 

int  x; 

switch (randomization^rule) 

{ 

fprintf  (fptr,**  \nrandomizat  ion  rule  is  "); 
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case  1: 

fprintf (fptr,*'  load  from  separate  files”); 
break; 

case  2: 

fprintf (fptr,”  load  from  single  files”); 
break; 

case  3: 

fprintf  (fptr,  "load  by  class  ”); 
for  (x  =  1;  X  <  classes  +  1;  x++) 
fprintf  (fptr,  ”\n  training  patterns  in  class  Xd  =  %d", 
X ,  training.patterns.in.class [x] ) ; 

} 

fprintf (fptr, ”\n"); 


/*  Function  Name:  file.seeds  Humber:  6.3  */ 
/*  Description:  This  function  files  the  seeds  used  to  randomly  set  */ 
/♦  network  parameters  and  load  the  data.  */ 
/♦  ♦/ 
/*  Functions  Called:  Hone  */ 
/*  Variables  Pass 3d  In;  wght.seed  -  Unsigned  ♦/ 
/*  sigma.seed  -  Unsigned  */ 
/♦  data^seed  -  Unsigned  ♦/ 
/*  record.seed  -  Unsigned  ♦/ 
/♦  ♦fptr  -  File  pointer  ♦/ 
/♦  ♦/ 
/♦  Variables  Rclurned:  lone  ♦/ 
/♦  Date:  11  Hov  90  Revision:  1.0  ♦/ 


/^**t***************i^**i^***t***i^***4t***********i^^i¥**^^****i^**t*******/ 

void  file.seeds (unsigned  wght.seed, 
unsigned  sigma.seed, 
unsigned  data.seed, 
unsigned  record.seed, 

FILE  ♦fptr) 


{ 

fprintf  (fptr,  ”\nwoight  seed  =  7.u  sigma  seed  =  y.u  data  seed  =  y,u  record  seed  =  y.u", 
wght.seed,  sigma.seed,  data.seed,  record.seed); 
fprintf (fptr , "\n”) ; 

} 


/♦  Function  Hame:  file.net.topology  Humber:  6.4  ♦/ 

/♦  Description:  This  function  files  the  topology  of  the  network  ♦/ 
/♦  such  as  feedforward  with  number  of  layers  ♦/ 

/*  */ 

/♦  Functions  Calles:  Hone  ♦/ 

/♦  Variables  Passed  In:  notwork.type  -  Integer  ♦/ 

/♦  number.of.no des  -  Integer  ♦/ 

/♦  nodes.in.layer  -  Integer  array  ♦/ 

/♦  ♦fptr  -  File  pointer  ♦/ 

/♦  ♦/ 

/♦  Variables  Returned:  Hone  ♦/ 

/♦  Date:  11  Hov  90  Revision:  1.0  ♦/ 


void  file.net.topology (int  network.type, 

int  number.of. layers, 
int  nodes.in.layer [] , 
FILE  vfptr) 
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int  x; 

if  (network^type  ==  1) 

fprintf  (fptr,"\n  network  type  =  feedforward  with  number  of  layers  =*  y.d**, 

number^of.layers) ; 

for  (x  «  0;  X  <  number.of «layers+l ;  x++) 
fprintf (fptr,”\n  nodes  in  layer  y*d  =  yd”,x,  nodes»in^layer[x]) ; 
fprintf  (fptr,''W') ; 


} 


/*  Function  lame:  f ile.tranfer.functions  Humber:  6.5  */ 
/*  Description:  This  function  files  the  transfer  function  for  each  */ 
/*  node  in  the  network.  */ 
/♦  */ 
/*  Functions  Called:  lone  */ 
/♦  Variables  Passed  In:  network^type  -  Integer  ♦/ 
/♦  number^of^layers  -  Integer  */ 
/*  start ing.node.in^layer  ~  Integer  array  ♦/ 
/♦  Hode.record  -  Structure  array  ♦/ 
/*  efptr  -  File  pointer  ♦/ 
/*  ♦/ 
/*  Variables  Returned:  lone  */ 
/♦  Date:  11  Hov  90  Revision:  1.0  ♦/ 


void  file^transfer.f  unctions  (int  network.type , 

int  number^ of .layers, 
int  starting.node_in.layer[] , 
struct  lode.data  vnode.record[] , 
FILE  efptr) 


int  x; 

if  (network.type  ==1) 

for  (x  =  0;  X  <  number.of .layers+1 ;  x++) 

switch (node .rccord[starting.node.in_layer Cx]]-> transfer. function) 

{ 

case  1: 

fprintf (fptr,''\n  layer  y,d  transfer  function  =  sigmoid*', x); 
break; 

case  2: 

fprintf (fptr,"\n  layer  y.d  transfer  function  =  rbf",x); 
break; 

case  3: 

fprintf (fptr,"\n  layer  ‘/d  transfer  function  =  linear", x); 
break; 

default : 
break; 

} 

} 

fprintf (fptr,"\n"); 


/♦  Function  Bame:  file.nodcs.at.data_points.info  Humber:  6.6  */ 
/*  Description:  This  function  files  the  par^eters  used  to  train  */ 
/*  a  layer  of  nodes  using  the  nodes.at.data.points  ♦/ 
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/♦  algorithm  ♦/ 
/*  ♦/ 
/*  Functions  Called;  Bone 

/♦  Variables  Passed  In:  current  .layer  -  Integer  ♦/ 
/*  output.threshold  -  Float  ♦/ 
/♦  sigma.threshold  -  Float  ♦/ 
/*  ♦fptr  -  File  pointer  ♦/ 
/♦  ♦/ 
i*  Variables  Returned:  Bone  */ 
/*  Date:  11  Bov  90  Revision:  1.0  ♦/ 


void  file.nodes_at.data.points.info(int  layer » 

float  out  put.  threshold, 
float  sigma.threshold, 
FILE  ♦fptr) 


{ 


} 


fprintf (fptr,“\nlayer  Xd  nodes  at  the  data  points",layer) ; 
fprintf (fptr,''\n  output  threshold  =  Xf  sigma  threshold  =  Xf*'> 

output .threshold,  sigma. threshold) ; 


/♦  Function  lame:  file.center.at.class.avgs.data  Bumber:  6.7  ♦/ 
/♦  Description:  This  function  files  the  parameters  used  to  train  ♦/ 
/♦  a  layer  of  nodes  using  the  center.at.class.avgs  ♦/ 
/♦  algorithm  ♦/ 
/♦  */ 
/♦  Functions  Called:  Bone  ♦/ 
/♦  Variables  Passed  In:  current  .layer  -  Integer  ♦/ 
/*  average.threshold  -  Float  ♦/ 
/♦  sigma.threshold  -  Float  ♦/ 
/♦  ♦fptr  -  File  pointer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  Bone  ♦/ 
/♦  Date:  11  Bov  90  Revision:  1.0  ♦/ 


void  file.center.at.class.avgs.data(int  layer, 

float  average.threshold, 
float  sigma.threshold, 
FILE  ♦fptr) 

{ 


> 


fprintf (fptr, "\n  layer  Xd  center  at  class  avgs'',layer) ; 
fprintf  (fptr,  "\n  average  threshold  =  Xf  sigma  threshold  =  Xf" 
average.threshold,  sigma.threshold) ; 


/♦  Function  Bame :  file_k.means.dat a  Bumber:  6.8  ♦/ 
/♦  Description:  This  function  files  the  parameters  used  to  train  ♦/ 
/♦  a  layer  of  nodes  using  the  k.means.cluster  ♦/ 
/♦  algorithm  ♦/ 
/*  ♦/ 
/♦  Functions  Called:  Hone  ♦/ 
/♦  Variables  Passed  In:  current.layer  -  Integer  ♦/ 
/♦  clusters  -  Integer  ♦/ 
/♦  ♦fptr  -  File  pointer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  Hone  ♦/ 
/♦  Date:  11  Bov  90  Revision:  1.0  ♦/ 


void  file_k.means.data(int  layer 


int  cl'JSterSf 
FILE  ♦fptr) 

/ 

fprintf (fptr>"\n  layer  %d  K  means  cluster", layer); 
fprintf (fptr,"\n  number  of  clusters  =  %d",  clusters); 

} 


/*  Function  lame:  file.kohonen.dat a  lumber:  6.9  */ 
/*  Description:  This  function  files  the  parameters  used  to  train  ♦/ 
/*  a  layer  of  nodes  using  the  train_Tia_kohonen  */ 
/♦  algorithm  ♦/ 
/♦  ♦/ 
/♦  Functions  Called:  lone  */ 
/♦  Variables  Passed  In:  current .layer  ~  Integer  ♦/ 
/*  nodes.x  ~  Integer  ♦/ 
/♦  nodes.y  -  Integer  ♦/ 
/♦  ♦fptr  ~  File  pointer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  lone  ♦/ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


/t4*************ii^*****************0**0*t^*^****t0******t*************/ 

void  file.kohonen.datadnt  layer, 

int  nodes.x, 
int  nodes.y, 

FILE  efptr) 

{ 

fprintf  (fptr,  "\nlayer%d  Kohoncn  Training" , layer) ; 
fprintf (fptr," \n  nodes  in  x  direction  =  Xd" , nodes.x) ; 
fprintf  (fptr,  "\n  nodes  in  y  direction—  y,d",  nodes.y); 

} 


/♦  Function  lame:  file.MSE.data  lumber:  6.10  ♦/ 
/♦  Description:  This  function  files  the  parameters  used  to  train  ♦/ 
/♦  a  the  remaining  layers  of  nodes  using  the  ♦/ 
/♦  MSE.remaining.layers  algorithm  ♦/ 
/*  ♦/ 
/♦  Functions  Called:  Hone  ♦/ 
/♦  Variables  Passed  In:  current.layer  -  Integer  ♦/ 
/♦  HSE. iterations  -  Integer  ♦/ 
/♦  MSE.error.delta  -  Float  */ 
/♦  MSE.momentum  -  Float  ♦/ 
/♦  HSE.successes  -  Integer  ♦/ 
/*  MSE.eta  ~  Float  ♦/ 
/♦  *fptr  -  File  pointer  ♦/ 
/*  ♦/ 
/♦  Variables  Returned:  Hone  ♦/ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


void  file.MSE_data(int  layer, 

int  MSE.iterations, 
float  MSE.error.delta, 
float  HSE.momentum, 
int  HSE.successes, 
float  nSE.ota, 

FILE  efptr) 

{ 

fprintf  (fptr," \nMSE  layer  %d  and  all  others", layer) ; 

fprintf  (fptr,"  \n  iterations  =  7,d  error  delta  =  y*f  momemtum  =  y.f", 

MSE.iterations,  MSE.error.delta,  MSE.momentum); 
fprintf  (fptr,  "\nsuccess  =  y,d  eta  =  y,f ", HSE.successes, MSE.eta) ; 
fprintf (fptr," \n") ; 
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} 


/♦  Function  Hama:  f ile.CFH^data  lumber:  6.11  */ 
/♦  Description:  This  function  files  the  parameters  used  to  train  ♦/ 
/*  a  the  remaining  layers  of  nodes  using  the  */ 
/♦  CFM«remaining.layers  algorithm  ♦/ 
/♦  ♦/ 
/♦  Functions  Called:  Hone  «/ 
/♦  Variables  Passed  In:  current «layor  ~  Integer  ♦/ 
/♦  CFM«alpha  -  Float  */ 
/♦  CFH.beta  -  Float  ♦/ 
/♦  CFM.eta  -  Float  ♦/ 
/♦  CFM.zeta  -  Float  ♦/ 
/♦  CFM.successes  -  Integer  >5^/ 
/♦  CFH. iterations  -  Integer  */ 
/♦  CFN.momentum  -  Float  ♦/ 
/♦  CFM.delta  ~  Float  ♦/ 
/♦  ♦fptr  -  File  pointer  ♦/ 
/♦  ♦/ 
/*  Variables  Returned:  Hone  a/ 
/♦  Date:  11  Hov  90  Revision:  1.0  ♦/ 


void  file«CFM«data(int  layer, 

float  CFH.alpha, 
float  CFM.beta, 
float  CFH.eta, 
float  CFH.zeta, 
int  CFM.successes, 
int  CFM. iterations, 
float  CFH.momentum, 
float  CFM.delta, 

FILE  ♦fptr) 

{ 

fprintf (fptr,”\nCFM  layer  ‘/d  and  all  others”, lay er) ; 
fprintf (fptr,  ”\nalpha  =  tf  beta  =  t±  eta  =  if  zeta  =  y.f", 

CFM.alpha,  CFM.beta,  CFM.eta,  CFM.zeta); 
fprintf  (fptr,  "\nsuccesses  =  y.d  iterations  =  ^d  momentum  =  iS  delta  =  y,f”, 
CFH^successcs,  CFH_ it orations,  CFH.momentum,  CFM.delta); 
fprintf (fptr , ”\n” ) ; 

> 


/♦  Function  Hame:  file«CE«data  Humber:  6.12  ♦/ 
/♦  Description:  This  function  files  the  parameters  used  to  train  ♦/ 
/♦  a  the  remaining  layers  of  nodes  using  the  ♦/ 
/♦  CE^remaining.layers  algorithm  ♦/ 
/♦  ♦/ 
/♦  Functions  Called:  Hone  ♦/ 
/♦  Variables  Passed  In:  current flayer  -  Integer  ♦/ 
/♦  CE.epsilon  -  Float  ♦/ 
/♦  MSE^it orations  -  Integer  ♦/ 
/♦  MSE.momentum  -  Float  ♦/ 
/♦  MSE.eta  -  Float  ♦/ 
/♦  HSE^successes  -  Integer  ♦/ 
/♦  *fptr  -  File  pointer  ♦/ 
/*  ♦/ 
/♦  Variables  Returned:  Hone  ♦/ 
/♦  Date:  11  Hov  90  Revision;  1.0  ♦/ 


void  file_CE«data(int  layer, 

float  CE_ epsilon, 
int  CE.iterations, 
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float  CE^nomentum, 
float  CE.eta, 
int  CE.successes, 

FILE  ♦fptr) 

{ 

fprintf (fptr,"\iiCE  layer  y.d  and  all  others”,l®yeJ^) » 
fprintf (fptr,  ''\neplison  =  Xf  iterations  =  ^d  nomentum  =*  ^f", 
CE.epsilon,  CE.iterations,  CE.momentnm) ; 
fprintf  (fptr, ''\net  a  =  y,f  errors  =  y,d”, 

CE^eta,  CE.successes) ; 
fprintf  (fptr  ,”\n'*) ; 


/*  Function  lame:  file.matrix.data  lumber:  6*13  */ 
/*  Description:  This  function  files  the  parameters  used  to  train  */ 
/*  a  the  remaining  layers  of  nodes  using  the  */ 
/*  global.MSE.algorithm  ♦/ 
/♦  ♦/ 
/*  Functions  Called:  lone  */ 
/*  Variables  Passed  In:  current.layer  -  Integer  ♦/ 
/♦  ♦fptr  -  File  pointer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  lone  ♦/ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


void  file.matrix.data(int  layer, 
FILE  ♦fptr) 


{ 


} 


fprintf  (fptr, '•\nLayer  y*d  Linear  by  Matrix  Inversion” , layer) ; 


/♦  Function  lame:  file.parzen_vindow.dat a  lumber:  6.14  ♦/ 
/♦  Description:  This  function  files  the  parameters  used  to  train  ♦/ 
/♦  a  the  remaining  layers  of  nodes  using  the  ♦/ 
/♦  Pll.implementation  algorithm  ♦/ 
/♦  */ 
/♦  Functions  Called:  lone  ♦/ 
/♦  Variables  Passed  In:  lode.record  -  Structure  array  ♦/ 
/♦  nodes_in_layer  -  Integer  array  ♦/ 
/♦  starting.node^in.layer  -  Integer  array  ♦/ 
/♦  current_layer  -  Integer  ♦/ 
/♦  ♦fptr  “  File  pointer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  lone  ♦/ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


void  f ile.parzen.uindov.dat a (struct  lode.data  ♦node.record[] , 
int  nodes.in.layerC] , 
int  starting^node^in.layerC] , 
int  layer, 

FILE  ♦fptr) 


int  X,  y,  current .node,  previous.layer.nod6: 
int  total.nodes  =  0; 

fprintf  (fptr,  “XnParzen  vindow  for  layer  y.d*', layer) ; 
for  (x  =  0;  X  <  nodes.in.layer [layer] ;  x++) 

{ 

current .node  =  st art ing.node.in.layer [layer] +x; 
fprintf  (fptr,  "Xnnode  y,d  vith  class  Xd  connected  to:  \n”, 
current .node,  node.rccord[current.node]->class) ; 
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total.nodes  «  0; 

for  (y  “  0;  y  <  nodes.in^layerClayer-l] ;  y++) 

{ 

preTiou8.layer.nodQ  “  start ing.node.in.layer Clay er-l]+y; 
if  (node.record [current .node] ">connect [previous.layer.node]  ==1) 

{ 

fprintf (fptr,"Xd  '',preTiou8.1ayor.node) ; 
total.nodes  =  total.nodes'i‘1 ; 

} 

} 

fprintf (fptr,“\n  total  nodes  for  this  node  is  Xd  total.nodes); 

} 


/«  Function  lame:  file.sigma.dat a  lumber:  6.15  ♦/ 
/«  Description:  This  function  files  the  parameters  used  to  train  */ 
/*  a  the  sigmas  of  an  RBP  notvork  by  any  of  the  ♦/ 
/♦  folloving  algorithms  ♦/ 
/♦  a)  scale.according.to.interference  ♦/ 
/♦  b)  set.sigmas.to.constant  ♦/ 
/*  c)  set.sigma.at.P.neighbor.avg  ♦/ 
/♦  */ 
/t  Functions  Called:  lone  t/ 
/♦  Variables  Passed  In:  current. layer  -  Integer  */ 
/♦  sigma.rule  -  Integer  */ 
/♦  interference. threshold  -  Float  ♦/ 
/♦  '  sigma.factor  -  Float  ♦/ 
/♦  sigma.constant  -  Float  ♦/ 
/♦  p.neighbors  -  Integer  ♦/ 
/♦  efptr  -  File  pointer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  lone  ♦/ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


void  file.sigma.dataCint  layer, 

int  sigma.rule, 

float  interference. threshold, 

float  sigma.factor, 

float  sigma.constant, 

int  p.neighbors, 

FILE  vfptr) 


fprintf (fptr,”\nsigmas  in  layer  Xd", layer); 
svitch(sigma.rule) 

{ 

case  1: 

fprintf (fptr,"\nsigmas  scaled  by  constant"); 
fprintf (fptr," interference  threshold  =  Xf  sigma  factor  =  Xf", 
interference.threshold,  sigma.factor) ; 

break; 
case  2: 

fprintf  (fptr,  "\nerror,  no  training  rule"); 
break; 

case  3: 

fprintf  (fptr,  "\nsigmas  set  to  a  constant"); 
fprintf  (fptr,"  sigma  constant  =  Xf ",  sigma.constant) ; 
break; 
case  4: 

fprintf (fptr, "\n  P  neighbors"); 

fprintf  (fptr,"  p.neighors  =  Xd",p.neighbors) ; 

break; 
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default : 
break; 

> 

fprintf<fptr,”\n»); 

} 


/*  Function  Kane:  print.last.layer.output  lumber:  6.16  */ 
/*  Description:  This  function  prints  the  outputs  of  the  last  e/ 
/♦  layer  of  the  network,  giving  the  node  number  ♦/ 
/*  node  output.  ♦/ 
/♦  ♦/ 
/♦  Functions  Called:  lone  ♦/ 
/«  Variables  Passed  In:  lode.record  **  Structure  array  e/ 
/*  nodes.in^layer  -  Integer  array  ♦/ 
/♦  start ing^node.in^layer  -  Integer  array  ♦/ 
/♦  last.layer  -  Integer  */ 
/♦  */ 
/*  Variables  Returned:  lone  ♦/ 
/«  Date:  11  Vov  90  Rovision:  1.0  «/ 


void  print.last.layer.output (struct  Kode.data  ♦node^recordC] , 

int  nodes.in.layerC], 

int  starting.node.in.layer[] , 

int  last.layer) 

{ 

int  X,  last.layer.nod6; 

for  (x  s*  0;  X  <  nodes.in.layerClast.layer] ;  x++) 

{ 

last .lay er.node  =  start ing.node.in.layer Clast .layer]  +  x; 
printf  ('*\n  node  Xd  output  =  Xf">last.layer.node 

,node.record [last.lay er.node] “>output) ; 

} 

> 


/♦  Function  Kane:  f ile.last.layer.output  Kumber:  6.17  ♦/ 
/♦  Description:  This  function  files  the  outputs  of  the  last  ♦/ 
i*  layer  of  tho  network,  giving  the  node  number  «/ 
/*  node  output,  data  record  number  and  class.  «/ 
/♦  ♦/ 
/«  Functions  Called:  Vone  */ 
/♦  Variables  Passed  In:  training  or  test.dat a  -  Structure  array  ♦/ 
/*  lode.record  -  Structure  array  ♦/ 
/♦  record  -  Integer  ♦/ 
/♦  nodes.in.layer  -  Intege..  array  ♦/ 
/♦  start ing.node.in.layer  -  Integer  array  ♦/ 
/*  last.layer  -  Integer  ♦/ 
/♦  ♦fptr  -  File  pointer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  Kone  ♦/ 
/♦  Date:  11  lov  90  Revision:  1.0  v/ 


void  file.last.Jayer.output (struct  data  vdata.recordE] , 

struct  Kode.data  ♦node.recordD , 

int  record, 

int  nodes.in.laycrC] , 

int  start ingjtiode.in_layer  □ , 

int  last.layer, 

FILR  ♦fptr) 

int  X,  last_layer.node; 
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fprintf (fptr/'\n\n  element  %d  ", record); 

fprintf  (fptr,"is  data  record  %d  with  class  7.d",data«record [record] ->nunbQr 

,data«record[record]“>class) ; 
for  (x  =  0;  X  <  node8.in.la7er[last.layer] ;  x++) 

{ 

last .lay er.node  «  start ing.node.in.layer [last.] ayer]  +  x; 
fprintf (fptr,"\n  node  Xd  output  =  %f",last.layer.nodo 

,nodo.record [last.layer.node] ->output ) 

> 


/o  Function  lame:  print  .data  lumber:  6.18  */ 
/*  Description:  This  function  prints  the  data  of  the  input  •/ 
/♦  files  as  read  by  the  software  */ 
/♦  ♦/ 
/*  Functions  Called:  lone  ♦/ 
/♦  Variables  Passed  In:  training  or  test. data  -  Structure  array  ♦/ 
/♦  train.set  or  test.set  ~  Integer  ♦/ 
/♦  dimension  -  Integer  ♦/ 
/*  */ 
/*  Variables  Returned:  lone  */ 
/♦  Date:  11  lov  90  ReTision:  1.0  ♦/ 


void  print .dataCstruct  data  edata.recordG , 
int  record.no, 
int  dimension) 


int  X,  y; 

for  (x  =0;  X  <  record.no;  x++) 

{ 

for  (y  =  0;  y  <  dimension;  y++) 

printf ("  )ilf",data.record[x]~>vector[y]) ; 
printf("\n  Xd  \n\n  ",  data.record[x]'->class) ; 

} 

} 

/*******i^***t**t**************t*****4*******************************/ 


/*  Function  leime:  file.dat a  lumber:  6.19  */ 
/*  Description:  This  function  files  the  data  of  the  input  */ 
/♦  files  as  read  by  the  software  ♦/ 
/*  */ 
/*  Functions  Called:  lone  */ 
/*  Variables  Passed  In:  training  or  test.data  -  Structure  array  */ 
/♦  train.set  or  test.set  -  Integer  ♦/ 
/♦  dimension  -  Integer  */ 
/*  ♦fptr  -  File  pointer  ♦/ 
/♦  ♦/ 
/*  */ 
/♦  Variables  Returned:  lone  ♦/ 
/♦  Date:  11  lov  90  Revisi'>n:  1.0  */ 


void  file.data(struct  data  ♦data.record[] , 
int  record.no « 
int  dimension, 

FILE  ♦fptr) 

{ 

int  X,  y; 

for  (x  =  0;  X  <  record.no;  x++) 

{ 

for  (y  =  0;  y  <  dimension;  y++) 
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fprintf (fptr,"  Xf”,data.rccordCx]“>Toctor[y]) ; 
fprintf  (fptr,'‘\n  Xd  \n  \n  dava«r€cordCx3“>class); 

} 

> 


/«  Function  lame:  print.no de.dat a  luabar:  6.20  */ 
/a  Description:  This  function  prints  the  data  structure  of  each  */ 
/*  node  in  the  network  including  the  weights,  */ 
/*  signas,  and  the  transfer  functions  */ 
/*  */ 
/*  Functions  Called:  lone  */ 
/♦  Variables  Passed  In:  lode.record  -  Structure  array  ♦/ 
/♦  current.node  -  Integer  ♦/ 
/*  total^nodes  -  Integer  ♦/ 
/*  ♦/ 
/*  Variables  Returned:  Vone  e/ 
/e  Date:  11  low  90  Revision:  1.0  e/ 


void  print.node.dataCstnict  k^^e.data  enode.recordC] , 
int  curront.node, 
int  total.nodes) 


int  x; 

printf ("\ndata  for  node  Xd",cttrrent.node); 
for  (x  =  0;  X  <  total.nodes;  x++) 
if  (node.record[current.node]->connect[x]  ==  1) 
printf ("\nweight  %d  »  %f'‘,x,node_record[current.node]“>weight[x]) ; 

if  (node.record[current.node3‘‘>transfer.function  ==2) 
for  (x  =  0;  X  <  total.nodes;  x++) 

{ 

if  (node.record [current .node3~>connectCx3  ==  1) 
printf (*’\nsigma  *  %f‘',node.recordCcttrrcnt_node3*->signia[x3) ; 

} 

else 

printf (‘*\n  sigma  y,d  =  Xf'jC^J^roxit.node 

,node.record [current .node3 ->sigma[current_node3 ) ; 
printf  (”\ntransfer  function  =  y»d“,node.record[current.node3">tremsfer_function) ; 


/♦  Function  Fame:  file.nodo.data  lumber:  6.21  */ 
/♦  Description:  This  function  files  the  data  structure  of  each  */ 
/*  node  in  the  network  including  the  weights,  e/ 
/*  sigmas,  eoid  the  transfer  functions  e/ 
/*  ♦/ 
/♦  Functions  Called:  lone  ♦/ 
/♦  Variables  Passed  In:  lode.record  -  Structure  array  ♦/ 
/♦  current.node  -  Integer  ♦/ 
/♦  total.nodes  -  Integer  ♦/ 
/♦  ♦fptr  “  File  pointer  ♦/ 
/*  */ 
/♦  Variables  Returned:  lone  ♦/ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


/«**«*««***i»«*«««*«*«***4t«*«««**4r]»*«4i««4t**«««**«*«***«*t  ♦♦♦♦*♦*«♦♦♦♦/ 

void  f  ile_node.data(struct  lode.data  ♦node_record[3  , 
int  current .node, 
int  total.nodes, 

FILE  ♦fptr) 

{ 

int  x; 
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fprintf (fptr,”\ndata  for  node  )id”,cnrrent.node) ; 
for  (x  «  0;  X  <  total.nodes;  x++) 

{ 

if  (nodo.rocord[current.iiode]->connectCx]  ==  1) 

{ 

fprintf (fptr,"\n  wght  /(d  *  %f  ",x,node^recordEcurrent«node]“>weightCx]) ; 
if  (node«recordCcurrent«node3->transfer.ftmctioii  *==  2) 

fprintf (fptr,**  sigma  74  =  tf  '’,x,nod0«rocord[current«node]">sigmaCx]) ; 

} 

} 

if  (node.record[ciirrerit.node]->transfer.function  ==  1) 
fprintf  (fptr,  "Xnsigwa  Xd  =  y,f",current«nodo 

,node«record[current«node]->siginaCcurrent^node]) ; 

fprintf  (fptr,  "Xntranffer  function  =  Xd”,node»record[current«nodej->transfer.function); 

} 


/♦  Function  lane:  print ^node.veights  lumber:  6.22  */ 
/♦  Description:  This  function  prints  the  weights  for  each  ♦/ 
/«  node  in  the  network.  */ 
/*  */ 
/*  Functions  Called:  lone  */ 
/♦  Variables  Passed  In:  lode.record  -  Structure  array  */ 
/♦  current .node  -  Integer  */ 
/*  total.nodes  -  Integer  ♦/ 
/♦  ♦/ 
/*  Variables  Returned:  lone  */ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


/^t**4^*******^*****t^*******t****t*********itit^***^****^i^***4*0**^***/ 

void  print. node.weights (struct  lode.data  ’^node.recordC] , 
int  current .node, 
int  total.nodes) 

{ 

int  x; 

printf  ("Xnlode  74",  current  .node) ; 
for  (x  =  0;  X  <  total.nodes;  x++) 

if  (node.recordCcurrent.node]*>connect [x]  ==  1) 
printf ("Xnweight  74  =  7.I",x,node.record[current_node]->weightCx]> ; 

} 


/*  Function  lame:  file.node.weights  b  lumber:  6.23  */ 
/*  Description:  This  function  files  the  weights  for  each  */ 
/*  node  in  the  network.  */ 
/♦  ♦/ 
/*  Functions  Called:  lone  */ 
/*  Variables  Passed  In:  lode.record  -  Structure  array  */ 
/♦  current.node  -  Integer  ♦/ 
/♦  total.nodes  -  Integer  ♦/ 
/*  ♦fptr  -  File  pointer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  Hone  ♦/ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


/4*0t*******ilf**********i^******t****tt****t***********t¥t***4***^*****/ 

void  file.node.weights (struct  lode.data  ♦node.rccordC] , 
int  current.node, 
int  total.nodes, 

FILE  ♦fptr) 

{ 

int  x; 

fprintf (fptr, "Xnlode  74", current .node) ; 
for  (x  =  0;  X  <  total.nodes;  x++) 
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if  (node.r€cord[current.node]->connect[x3  ==  1) 
fprintf  (fptr/*\nweight  %d  =  y,f”,x,node_recordCciirrent_node3->weightCx3) ; 

} 


/*  Function  lame:  print.node.sigma  Humber:  6.24  */ 
/*  Description:  This  function  prints  the  sigmas  for  each  >>/ 
/*  node  in  the- network.  */ 
/*  */ 
/*  Functions  Called:  Hone  */ 
/♦  Variables  Passed  In:  Kode^record  -  Structure  array  ♦/ 
/♦  current .node  -  Integer  ♦/ 
/♦  total.nodes  -  Integer  ♦/ 
/*  ♦/ 
/*  Variables  Returned:  Hone  */ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


/^t******t*****************^i^*******t*******i^*^***^^***i^iti**i¥^^**^***/ 

void  print.node.sigmaCstruct  lode.dat a  ♦node.recordO  , 
int  current .node, 
int  total.nodes) 

{ 

int  x; 

printf  (''\nIode  y«d*',  current  .node) ; 
if  (node.record [current . node] “>transfer.funct ion  ==  2) 
for  (x  =  0;  X  <  total.nodes;  x++) 

{ 

if  (node.r e cord [curr cut. node] ~>connect[x]  ==  1) 
printf  ("\n  sigma  y,d  =  y,f",x,  no  de.re  cord  [current. no  de] ->s  igma  [x]  ) ; 

} 

else 

printf  (“Vn  sigma  y.d  =  y,f”,  current  .node, 

node.record[current.node]->sigma[current.node] ) ; 


} 


/♦  Function  lame:  print.node.output  Humber:  6.25  ♦/ 
/♦  Description:  This  function  prints  the  output  for  a  given  ♦/ 
/*  node  in  the  network.  ♦/ 
/♦  */ 
/♦  Functions  Called:  Hone  */ 
/*  Variables  Passed  In:  Hode.record  Structure  array  */ 
/*  current.node  -  Integer  ♦/ 
/*  ♦/ 
/♦  Variables  Returned:  Hone  */ 
/♦  Date:  11  Hov  90  Revision:  1.0  */ 


/*t*i^*i^*******^********’¥*********ilf************t****t***^t****t*t****/ 

void  print.node.output (struct  Kode.data  tnode.recordn , 
int  current.node) 


printf  ("\n  Hodo  y,d  output  =  Xf',  current  .node,  no  do.re  cord  [curr  ent. node] ->  out  put ) ; 

> 


/♦  Function  Name:  print  .node  .transfer.f  unction  Humber:  6.26  ♦/ 

/♦  Description;  This  function  prints  the  transfer  function  for  a  */ 


given  node  in  the  network.  */ 
/♦  ♦/ 
/♦  Functions  Called:  Bono  ♦/ 
/*  Variables  Passed  In:  Hode.record  -  Structure  array  */ 
/*  current.node  *•  Integer  ♦/ 
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/*  total.nodes  -  Integer  ♦/ 
/*  ♦/ 
f*  Variables  Returned:  None  */ 
/♦  Date:  11  Nov  90  Reyision:  1.0  ♦/ 


void  print^node«transfer.function(stnict  lode.data  vnode.rocordC]  , 

int  current .node) 


printf('*\n  Ho'<  *{i  transfer  function  =  y«d",cttrr®i^t_node,node.record[current.node]->transfer.function) ; 

} 


/*  Function  Name:  file.network.paraffleter8  Number:  6.27  */ 
/*  Description:  This  function  files  the  network  parameters>  */ 
/♦  network  type,  number  of  layers,  nodes  in  each  */ 
/*  layer,  node  transfer  functions,  training  rules  ♦/ 
/♦  and  the  interconnection  topology  for  the  network  ♦/ 
/♦  nodes  ♦/ 
/♦  ♦/ 
/*  Functions  Called:  None  ♦/ 
/*  Vari:  ^*es  Passed  In:  Node.record  -  Structure  array  ♦/ 
/*  network.type  -  Integer  ♦/ 
/♦  number. of. layer  -  Integer  ♦/ 
/♦  nodes.in.layer  ~  Integer  array  ♦/ 
/♦  training.rule  -  Integer  array  ♦/ 
/♦  transf er.f unction  -  Integer  array  ♦/ 
/♦  sigma.rule  -  Integer  ♦/ 
/♦  total.nodos  -  Integer  ♦/ 
/♦  *fptr  -  File  pointer  ♦/ 
/♦  ♦/ 
/*  Variables  Returned:  None  ♦/ 
/♦  Date:  11  Nov  90  Revision:  1.0  */ 


void  file.network.parameters (struct  Node.data  ^node.recordC] , 

int  network.type, 
int  number.of. layers, 
int  nodes.in.layer [] , 
int  training.rule [] , 
int  t ransf er.f unct ion C3 , 
int  sigma.rule, 
int  total.nodos, 

FILE  ♦fptr) 

{ 

int  X,  y; 

fprintf (fptr,*‘\nNetwork  Parameters”) ; 
fprintf (fptr,”\nHetwork  type  =  */,d”,network.type) ; 
fprintf (fptr, ”\nNumber  of  layer  =  Xd”, number. of. lay ers ) ; 
for  (x  =  0;  X  <  number. of  .layers  +  1;  x++) 

{ 

fprintf  (fptr,' '\n  nodes  in  layer  y.d  =  y.d”,  x, nodes.in.layer  [x]  ) ; 
fprintf  (fptr,”  transfer  function  =  y.d”,  transf  er.f  unct  ion  [x]  ) ; 
fprintf  (fptr,”  training  rule  =  y,d",training.rule[x3) ; 

} 

fprintf  (fptr,  ”\n  sigma  rule  =  y.d”,sigma.rule) ; 
for  (x  =  0;  X  <  total.nodes;  x++) 

fprintf  (fptr,  "\n\n  node  ‘/.d  receives  input  from  the  following  nodes\n",x); 
for  (y  =  0;  y  <  total.nodcs;  y++) 

{ 

if  (node.record [x3->connect[y3  ==  1) 
fprintf  (fptr,  "'/.d  ”,y); 

> 
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} 

} 


/*  Function  lame:  file.error.data  lumber:  6.28  */ 
/*  Description:  This  function  files  the  total  number  ofch  */ 
/♦  errrors,  the  percentage  correct,  and  the  record  ♦/ 
/♦  misclassified  ♦/ 
/*  */ 
/♦  Functions  Called:  Hone  ♦/ 
/♦  Variables  Passed  In:  class.error  -  Integer  t/ 
/*  per^cent^correct  -  Float  ♦/ 
/♦  misclassifier  ~  Integer  Array  ♦/ 
/♦  ♦fptr  -  File  pointer  ♦/ 
/♦  ♦/ 
/*  Variables  Returned:  Hone  */ 
/♦  Date:  11  Hov  90  Revision:  1,0  ♦/ 


void  file«error.data(int  class.error, 

float  per«cent_correct , 
int  misclassified [] > 

FILE  ♦fptr) 

{ 

int  x; 

fprintf (fptr,’'\n  total  errors  =  y.d",class^error) ; 
fprintf (fptr,"\n  per  cent  correct  =  yf“,per«cent«correct) ; 
for  (x  =  0;  X  <  class^error;  x++) 
fprintf  (fptr,  “\n  record  %d  misclassified  ",misclassified[x]) ; 

} 


/*  Function  Hame:  file.class.count  Humber:  6.28  ♦/ 
/*  Description:  This  function  files  the  number  of  training  */ 
/♦  patterns  from  each  class  ♦/ 
/♦  ♦/ 
/♦  Functions  Called:  Hone  ♦/ 
/*  Variables  Passed  In:  training«patterns^in.class  -  Integer  array  */ 
/♦  classes  -  Integer  ♦/ 
/♦  ♦fptr  -  File  pointer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  Hone  ♦/ 
/♦  Date:  11  Hov  90  Revision:  1.0  ♦/ 


void  file_class_count  (int  training^patterns^in.classC] , 
int  classes, 

FILE  ♦fptr) 


{ 

int  x; 

for  (x  =  1;  X  <  classes  +1;  x++) 

fprintf  (fptr,  "\n  records  in  class  */.d  =  yd",x,  training_pattems_in.class[x]) 

> 
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G.7  NETOUT 


/^*t************************************t****4i****t*0****tt*********/ 

/«  Module  lame:  lETOUT.C  lumber:  7.0  */ 

/*  Description:  This  module  contains  the  functions  uhich  */ 

/♦  calculate  the  output  for  each  node  in  the  network  ♦/ 

/♦  due  to  a  given  input  pattern  ♦/ 

/*  ♦/ 

/♦  Modules  Called:  lone  */ 

/♦  Functions  Contained:  7.1  calculate.feed«forward^network_output  ♦/ 

/♦  7.2  calculate_layer«0^output  ♦/ 

/♦  7.3  cal culate.layer.l .output  ♦/ 

/♦  7.4  calculate.layer_2.output  */ 

/*  7.5  calculato.layer.3.output  */ 

/*  7.6  calculate.nodo.output  ♦/ 

/♦  7.7  calculate.output.as. input  ♦/ 

/♦  7.8  calculate.linear.output  ♦/ 

/♦  7.9  calcul at e.rbf. output  ♦/ 

/♦  7.10  calculate. sigmoid. output  */ 

/*  */ 

/*  Date:  11  lov  90  Revision:  1.0  ♦/ 

tinclude"netvrble  .h'* 

#include"netf nctn .  h*' 

/*  Function  lame:  calculate.feed.forward.network.output  lumber :7.1*/ 

/*  Description:  This  function  calculates  the  output  of  a  feed  */ 

/*  forward  network  due  to  an  input  pattern  ♦/ 

/♦  V 

/♦  Functions  Called:  7.2  calculate. lay er.O. output  ♦/ 

/♦  7.3  calculate.layer.l. output  */ 

/*  7.4  calculate.layer.2.output  ♦/ 

/♦  7.5  calculate.layer.3.output  ♦/ 

/♦  ♦/ 

/♦  Variables  Passed  In:  training  or  test.data  -  Structure  array  */ 

/♦  Hode.record  ~  Structure  array  ♦/ 

/*  number.of.laycrs  -  Integer  */ 

/♦  nodes.in.layer  -  Integer  array  ♦/ 

/♦  ♦/ 

/*  Variables  Returned:  Hode.record  -  Structure  array  ♦/ 

/♦  Date:  11  Hov  90  Revision:  1.0  ♦/ 

void  calculate.fecd.forward.network.output  (struct  data  ♦data.recordQ , 

struct  Vode.data  ♦node.recordQ , 

int  number.of. layers, 

int  nodes.in.layer  □ , 

int  starting.node.in.layerC] , 

int  record, 

int  total.nodes) 

{ 

int  layer; 

for  (layer  =  0;  layer  <  number.of.laycrs  +  1;  layer++) 

{ 

switch  (layer) 

{ 

case  0: 

calcttlate.layer.0.output (dat a.record , 
node.record, 
nodes.in.layer , 
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breaX; 


record) ; 


case  1: 

calculate«layer.l«output(data«record, 
nodo.record, 
nodes^in.layer , 

St  art ing^node^in_layer , 
total.nodes) ; 

break; 
case  2: 

calculate^layer_2_output (data^record> 
node.record, 
nodes.in.layer, 
starting_node_in«layer , 
total.nodes) ; 


break; 
case  3: 

calculate_layer_3_output(data.record, 
node.record, 
nodes_in_layer, 
st  art  iitg_node«in«lay  er , 
total.nodes) ; 

break; 
default : 

printf ("Xnerror'O ; 

} 

> 

} 

/^i^:i^^:titiilL04^T^t^i^t*********t*t*****t*****************4*if*0******4******/ 


/♦  Function  Vame:  calculate.layer.O^ output  Humber:  7.2  ♦/ 
/*  Description:  This  function  calculates  the  output  for  each  node  */ 
/*  in  layer  0  of  a  feed  forward  network  ♦/ 
/♦  */ 
/♦  Functions  Called:  7.6  calculate.node.output  ♦/ 
/♦  Variables  Passed  In:  training  or  test^data  -  Structure  array  ♦/ 
/.♦  Hode^record  -  Structure  array  ♦/ 
/♦  nodes.in.layer  -  Integer  array  ♦/ 
/»  record  -  Integer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  Vode.record  -  Structure  array  ♦/ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


void  calculate .lay er_0„out put  (struct  data  ♦data.recordC] , 

struct  Hode.dat a  ♦node.recordC] , 
int  nodes.in.layerC] , 
int  record) 


{ 

int  nodeO; 

for  (nodeO  =  0;  nodeO  <  nodes.in_layer[0] ;  nodeO++) 
calculat e_node.output  (data.record  > 
node.record, 
nodeO, 
record) ; 
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/♦  Function  lame;  calculate^layer^l.output  limber:  7.3  ♦/ 
/♦  Description:  This  function  calculates  the  output  for  each  node  ♦/ 
/♦  in  layer  1  of  a  feed  foxvard  network  ♦/ 
/♦  ♦/ 
/*  Functions  Called:  7.6  calculate_node^output  */ 
/♦  Variables  Passed  In:  training  or  test.data  -  Structure  array  ♦/ 
/♦  lode.record  -  Structure  array  ♦/ 
/♦  nodes«in«layer  -  Integer  array  ♦/ 
/♦  starting.node.in.layer  -  Integer  array  ♦/ 
/♦  total.nodes  -  Integer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  lode.record  -  Structure  array  ♦/ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


void  calculate.layer.l .output  (struct  data  ♦data.recordG , 

struct  Iode.dat a  enode.recordC] , 
int  nodes.in.layerC] > 
int  starting.node.layerQ , 
int  total.nodes) 

{ 

int  nodel , current .node ; 

for  (nodel  =  0;  nodel  <  nodes.in.layerCl];  nodel++) 

{ 

current.node  =  start ing.node.layerCl]  +  nodel; 
calculate.node.output  (data.record, 
node.record, 
current.node, 
total.nodcs) ; 

> 

} 

/****i^^^i^^^^t****i¥***********i^**************************************/ 


/*  Function  Hame:  calculate.layer.2.output  Number:  7.4  ♦/ 
/*  Description:  This  function  calculates  the  output  for  each  node  */ 
/*  in  layer  2  of  a  feed  forward  network  ♦/ 
/♦  ♦/ 
/♦  Functions  Called:  7.6  calculate.node.output  ♦/ 
/♦  Variables  Passed  In:  training  or  test.dat a  -  Structure  array  ♦/ 
/♦  Hode.record  ~  Structure  array  ♦/ 
/♦  nodes.in.layer  -  Integer  array  ♦/ 
/♦  start ing.node.in.layer  -  Integer  array  ♦/ 
/♦  total.nodes  -  Integer  ♦/ 
/♦  ♦/ 
/*  Variables  Returned:  Hode.record  -  Structure  array  ♦/ 
/*  Date:  11  Nov  90  Revision:  1.0  ♦/ 


/i^4*********^**t*it******0****T^*****t**flf**************t^*************i^/ 

void  calculate.layer.2.output  (struct  data  vdata.recordC] , 

struct  Node.dat a  ♦node.recordE] , 
int  nodes.in.layer C3 , 
int  starting_node_layerD , 
int  total.nodcs) 

{ 

int  node2,  current.node; 

for  (node2  =  0;  node2  <  nodes. in.laycrC2] ;  node2++) 

{ 

current .node  =  starting_node_layorC2]  +  node2; 
calculate.nodo.output  (data.record, 
node. record, 
current.node, 
total.nodes) ; 

} 
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} 


/*  Function  lame:  calculate.layer.S.output  lumber:  7.5  ♦/ 
/*  Description:  This  function  calculates  the  output  for  each  node  */ 
/*  in  layer  3  of  a  feed  forward  network  */ 
/♦  ♦/ 
/*  Functions  Called:  7.6  calculate_node.output  ♦/ 
/♦  Variables  Passed  In:  training  or  test.data  -  Structure  array  ♦/ 
/♦  lode^record  -  Structure  array  */ 
/*  nodes.in^layer  -  Integer  array  */ 
/♦  starting»node^in.layer  -  Integer  array  ♦/ 
/*  total.nodes  -  Integer  */ 
/*  */ 
/*  Variables  Returned:  Vode.record  ~  Structure  array  */ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


void  calculate.layer.S.output  (struct  data  vdata.record[] , 

struct  lode^data  *node.record[] , 
int  nodes.in«layerC] , 
int  starting.node.layerD » 
int  total.nodes) 


{ 

int  nodes,  current .node; 

for  (nodes  =  0;  nodeS  <  nodes.in.layerCS] ;  node3++) 

{ 

current  .node  =  starting.node.layer[33  +  nodoS; 
calculate.node.output  (data.record, 
node.record, 
current .node, 
total.nodes) ; 

} 

} 

/^^*^*Jtft***t^’$^*****************************^*^^l^t****t^**0**********/ 


/♦  Function  lame:  calculate.node.output  lumber:  7.6  ♦/ 
/♦  Description:  This  function  calculates  the  output  for  a  single  ♦/ 
/♦  node  in  a  network  by  testing  the  transfer  ♦/ 
/*  function  and  sending  control  to  the  appropriate  */ 
/♦  function  */ 
/♦  ♦/ 
/♦  Functions  Called:  7.7  calculate.output. as. input  ♦/ 
/♦  7.8  calculate.linear.output  ♦/ 
/♦  7.9  calculate.rbf. output  ♦/ 
/♦  7.10  calculate.sigmoid.output  ♦/ 
/♦  ♦/ 
/♦  Variables  Passed  In:  training  or  test.data  -  Structure  array  ♦/ 
/♦  Hode.record  -  Structure  array  ♦/ 
/♦  currcnt.node  -  Integer  array  ♦/ 
/*  total.nodes  -  Integer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  Hode.record  -  Structure  array  ♦/ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


void  calculate.node.output (struct  data  ♦data.recordC] , 

struct  Kode.data  «node.record[3 , 
int  current .node , 
int  total.nodes) 


{ 
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siiitch  (node«record[current«node3  ->transf er.function) 

{ 

case  0: 

calculate.output.as.inpat  (data.record, 

node.record, 
current .node I 
total.nodes) ; 

break; 
case  1  : 

calculate.s igmoid.output (node.record, 
current .node , 
total.nodes) ; 

break; 
case  2  : 

calculate.rbf.output(node.record> 
current.node , 
total.nodes) ; 

break; 
case  3  : 

calculate.linear.output (node.record, 

current.node > 
total.nodes) ; 

break; 
default : 
break; 

} 

} 

/*t^00:^^4*****************************4*4-t***********0**************/ 


/♦  Function  Hame:  calculate.output.as.input  lumber:  7.7  ♦/ 
/♦  Description:  This  function  calculates  the  output  for  node  */ 
i*  with  the  identity  transfer  function  as  ♦/ 
/♦  y(out)  =  x(in)  ♦/ 
/*  ♦/ 
/♦  Functions  Called:  Hone  */ 
/♦  Variables  Passed  In:  training  or  test. data  -  Structure  array  ♦/ 
/♦  lode.record  -  Structure  array  ♦/ 
/♦  current.node  -  Integer  ♦/ 
/♦  record  -  Integer  ♦/ 
/♦  */ 
/♦  Variables  Returned:  lode.record  -  Structure  array  ♦/ 
/*  Date:  11  Hov  90  Revision:  1.0  ♦/ 


void  calculate.output.as.input (struct  data  ♦data.recordC] , 

struct  Bode.data  ♦n.recordD , 
int  node, 
int  record) 

{ 

n.record[node]~>output  =  data.recordCrecord3->vectorCnode] ; 
n.record[node3~>class  =  data.record[rccord3->class; 

} 


/♦  Function  Hame:  calculate.linear.output  Humber:  7.8  ♦/ 
/♦  Description:  This  function  calculates  the  output  for  node  ♦/ 
/♦  with  the  linear  transfer  function  as  ♦/ 
/*  y(l)  =  sumCw(kl)x(k)3  ♦/ 
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/*  ♦/ 

/♦  Function:;  Called:  lone  ♦/ 
/*  Variables  Passed  In:  Vode.record  -  Structure  array  ♦/ 
/♦  current.node  -  Integer  ♦/ 
/♦  total.nodes  -  Integer  ♦/ 
/♦  */ 
/♦  Variables  Returned:  lode.record  -  Structure  array  ♦/ 
/*  Date:  11  Iot  90  ReTision:  1.0  */ 


void  calculate.linear^ output  (struct  lode.data  enode.record[] » 

int  current .node » 
int  total.nodes) 


int  node; 

node.record [current .node] ->output  =*  0; 
for (node  =  0;  node  <  total.nodes;  node++) 

{ 

node.recordCcurrent .node] ->output  =  node.rccord [cur rent .node] ->output 

+  node.record[curr6nt.node] “>weight [node] 

♦  node.record[nodo]->output 

*  node.record[current.node]*>connect [node] ; 


> 

} 


/♦  Function  Vane:  calculate.xbf. output  lumber:  7.9  */ 
/♦  Description:  This  function  calculates  the  output  for  node  ♦/ 
/e  uith  the  linear  transfer  function  as  */ 
/♦  y(l)  =  exp('-[l/2]sum{[x0c)~w(kl)]"2/«"2]  */ 
/*  */ 
/*  Functions  Called:  lone  */ 
/♦  Variables  Passed  In:  lode.record  -  Structure  array  ♦/ 
/♦  current.node  -  Integer  ♦/ 
/♦  total.nodes  -  Integer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  lode.record  -  Structure  array  ♦/ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


void  calculate.rbf. output (struct  lode.data  ♦nodo.record[] , 
int  current.node, 
int  total.nodes) 


{ 

double  buffer  =0; 
double  weight. off set  =  0; 
double  X  =  2.0; 
int  node; 

for  (node  =  0;  node  <  total.nodes;  nodc++) 

{ 

if (node.record[current.node]~>connect[node]  !=  0) 

{ 

weight.offset  =  (node.record[current.node]~>weight[node]- 
node.record[node]->output) ; 

buffer  =  buffer 

+  pow((weight.offset/node_record[current.node]->sigma[node]) ,x) ; 

} 

> 

node.record  [current .node] “>output  =  exp  (-(double) (buffer/2)); 


/*«««***«*««##***]»]»««*«*«*«* 
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/♦  Function  lame:  calculate.sigmoid.output  lumber:  7.10  ♦/ 
/*  Description:  This  function  calculates  the  output  for  node  */ 
/*  with  the  sigmoidal  transfer  function  as  ♦/ 
/♦  y(l)  “  l/{l+expC-suiaCw(kl)x(k)3+C]>  */ 
/♦  ♦/ 
/♦  Functions  Called:  lone  */ 
/♦  Variables  Passed  In:  lode.record  -  Structure  array  ♦/ 
/♦  current.node  -  Integer  ♦/ 
/♦  total.nodes  -  Integer  ♦/ 
/♦  ♦/ 
/e  Variables  Returned:  lode.record  -  Structure  array  «/ 
/♦  Date;  11  Iot  90  Revision:  1.0  ♦/ 


void  calculate.sigmoid.output (struct  lode^data  ♦node.recordD  | 

int  current .node, 
int  total.nodes) 

double  buffer  =  0; 
int  node; 

for  (node  =  0;  node  <  total.nodes;  nodo++) 

{ 

if  (nodo.record[current.node]->connect[node3  !=  0) 
buffer  s  buffer  +  node.r ecord [current _node3“> weight [node3 
♦  node.recordCnode3“>output; 

} 

buffer  =  buffer  +  node.record [current .node3">sigma [current _node3 ; 
node.record [current .node3“>output  =  1/(1  +  exp (-buffer)  ) ; 


G.8  NETAUX 


/*  Module  lame:  HETAUX  lumber:  8.0  ♦/ 

/♦  Description:  This  module  contains  the  training  functions  called  ♦/ 


/♦ 

by  lETTRAIl 

*/ 

/* 

♦/ 

/*  Modules  Called:  lone 

♦/ 

/♦ 

Functions  Contained:  8.1 

determine  Y  matrix 

♦/ 

/* 

8.2 

determine  S  matrix 

♦/ 

/* 

8.3 

determine  M  matrix 

*/ 

/♦ 

8.4 

calculate  weight  matrix 

♦/ 

/* 

8.5 

MSE.last .layer 

♦/ 

/* 

8.6 

MSE.last.layer_linear 

♦/ 

/* 

8.7 

MSE.last.layer.sigmoid 

♦/ 

/* 

8.8 

MSE.mid.layer 

*/ 

/* 

8.9 

MSE.lst.layer 

*/ 

/* 

8.10 

calculat  e.err ors.in.output 

h 

8.11 

get.linear.training.eta 

*/ 

/♦ 

8.12 

get.kohonen.neighborhood 

♦/ 

/♦ 

8.13 

calc.dist.outputs_to_nxt.lyr 

♦/ 

/* 

8.14 

find.nearest.elemcnt 

♦/ 

/* 

8.15 

find.kohonen.boundaries 

♦/ 

/* 

8.16 

dctermine.neighborhood.elements 

♦/ 

/* 

8.17 

train.kohonen. weights 

♦/ 

/* 

8.18 

f ind.distance.between.nodes 

*/ 

/* 

8.19 

sort_2_dim. array 

♦/ 

/♦ 

8.20 

CE.last.layer 

*/ 

/♦ 

8.21 

CE.mid.layer 

♦/ 

/♦ 

8.22 

CE.first. layer 

♦/ 
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/* 

8.23  calculate.zn 

♦/ 

/♦ 

8.24  CFM_last .layer 

*/ 

/* 

8.25  CFH.mid.layer 

♦/ 

/♦ 

8.26  find.se cond.highe8t.no de 

♦/ 

/* 

8.27  CFM.first.layer 

♦/ 

/* 

8.28  find.nearest.neighbor 

♦/ 

/♦ 

♦/ 

/♦  Date:  10  lov  90 

Revision:  1.0 

♦/ 

/«***«**«*«*«**«*«**«*«««#***#*««*«**«*«**«*«****«*♦*««#**«*«#****«]»/ 

iiticltide  "netvrble.h" 
iincltide  "netfnctn.h'' 

/^^tt**************t***************t**t****4**********4******t*0****/ 


/«  Function  lasie:  determine.Yjnatrix  lumber: 8.1  */ 
/*  Description:  This  function  calculates  the  outputs  for  each  */ 
/*  node  in  a  given  layer  and  stores  these  outputs  in  */ 
/*  a  matrix.  Each  rov  will  represent  the  outputs  for  */ 
/*  that  layer  due  to  an  input  pattern  *1/ 
/♦  Functions  Called:  calculate_layer_0«output  ♦/ 
/♦  calculate_layer_l«output  ♦/ 
/♦  calculate«layer^2_output  ♦/ 
/♦  calculatc«laycr«3_ output  ♦/ 
/♦  ♦/ 
/*  Variables  Passed  In:  lode.xecord  -  Structure  array  */ 
/♦  training.data  -  Structure  array  ♦/ 
/♦  train^set  -  integer  ♦/ 
/*  nodes^in^layer  -  integer  array  ♦/ 
/*  starting.node_in.layer  -  integer  array  ♦/ 
/♦  total.nodes  -  integer  ♦/ 
/♦  *Y  “  float  array  pointer  ♦/ 
/*  current flayer  -  integer  */ 
/*  */ 
/♦  Variables  Returned:  ♦Y  -  float  array  pointer  ♦/ 
/♦  Date:  10  Hot  90  Re vision: 1.0  ♦/ 


void  detennine.Y.matrix (struct  lode.data  ♦K^recordC], 
struct  data  ^data^recordD  > 
int  record^no, 
int  nodes.in«layer[] , 
int  starting_node_in^laycrC3, 
int  total.nodesy 
float  *Yn, 
int  current .layer) 


{ 

int  row,  column >  current .node; 

for  (row  =  0;  row  <  record.no;  row++) 

{ 

switch(current .layer) 

{ 

case  1 : 

calculate. lay  er.O.out  put  (data.record , 
M.record, 
nodes.in.layer, 
row) ; 

break; 
case  2: 

calculate.layer.O. output (data.record, 
H.record, 
nodes.in.layer, 
row) ; 
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calculate«layer.l«output(data.record, 

I.record, 

nod«s_in«layer, 

starting^node^in.layer, 

total.nodes); 

break; 
case  3: 

calculat  e^lay  er.O.outpiit  (data.record , 

I.record, 

nodes_in.layer> 

row); 

calculat  e .lay er^l ^output (data.r ecord , 

V.record» 
nodes.in.layer, 
start ing_node_in.layer, 
total.nodes) ; 

calculate.layer.2.output (data.record , 

H.record, 
nodes.in.layer, 
startiiig.node_in.layer, 
total.nodes) ; 

default : 

break; 

> 

for  (coluon  =  0;  column  <  nodes.in.layer[currcnt_layer-l] ;  column++) 

{ 

current .node  «  start ing.node.in.layer [current .layer  -  1]  +  column; 

♦  ( (Y  Crow] )+column)  =  I.record [current.node] ->output ; 

} 

} 

} 


/*  Function  Rame:  d6tennine.S.inatrix  lumber:  6.2  */ 
/*  Description:  This  function  sets  the  desired  outputs  for  the  */ 
h  last  layer  to  a  1  for  the  node  responsible  for  the  ♦/ 
!*  class  and  0  for  the  remaining  nodes  */ 
/♦  */ 
/♦  Functions  Called:  Rone  ♦/ 
/♦  Variables  Passed  In:  training.data  -  Structure  array  ♦/ 
/♦  train.set  -  Integer  ♦/ 
/♦  nodes. in.layer  -  Integer  array  ♦/ 
/♦  *5  -  Float  array  pointer  ♦/ 
/♦  current .layer  -  integer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  *S  array  ♦/ 
/*  Date:  10  Rov  90  Revision:  1.0  ♦/ 


void  determine.S.matrix (struct  data  ♦data^recordCl » 
int  record.no, 
int  nodes.in.layerC] , 
float  ♦S[], 
int  current.layer) 

i 

int  rov,  column; 

for  (rov  =  0;  row  <  record.no;  row  ++) 
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for  (column  =  0;  column  <  nodes.in.layerCcurrent.layer] ;  column++) 
♦((SCrow])+column)  =  0; 
for  (row  =*  0;  row  <  rocord.no;  row  ++) 

{ 

column  «  data.record[row3->class; 

♦  ((SCrow])+coltiain-l)  *  1; 

> 


/♦  Function  lame:  d«termin6.Hjnatrix  lumber:  8.3  */ 
/«  Description:  This  function  determines  the  cross  products  of  the  */ 
/*  outputs  from  all  the  nodes  in  a  given  layer  with  a  */ 
/*  specific  node  in  that  layer  ♦/ 
/♦  K(1B)  a  8umCy(pl)y(pB)]  */ 
/♦  Functions  Called:  lone  */ 
/♦  Variables  Passed  In:  *Y  -  Float  array  pointer  ♦/ 
/♦  ♦M  “  Float  array  pointer  ♦/ 
/♦  nodcs.in.layer  -  Integer  array  ♦/ 
/*  train.set  -  Integer  ♦/ 
/*  current.layer  -  Integer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  -  Float  array  pointer  ♦/ 
/*  Date:  10  lov  90  Revision:  1.0  */ 


void  deteraine.M.aatrix (float  *Y[], 
float  «MC3 , 
int  nodes.in.layer[3> 
int  patterns, 
int  current.layer) 

{ 

int  row,  coltuon,  p; 

for  (row  =  0;  row  <  nodes.in.layer  [current _layer-l3 ;  row++) 
for  (column  “0;  column  <  nodes.in.layer [current. lay er-l3  ;  column++) 

{ 

♦((M[row3)+column)  =  0; 

for  (p  =  0;  p  <  patterns;  p++) 

♦((M[row3)+column)  =  ♦((Y[p3)+row)  ♦  ♦((Y[p3)+column)  +  *((M[rov3)+column) 

} 


i*  Function  lame:  calculate.weightjnatrix  lumber:  8.4  */ 
/«  Description:  This  function  calculates  the  output  layer  weights  e/ 
/♦  by  w(BD)  =  sum{sum[y(pl)d(pD)3l(Bl)}  ♦/ 
/*  Functions  Called:  lone  *f 
/*  Variables  Passed  In:  lode.record  ~  Structure  array  ♦/ 
/♦  -  float  array  pointer  ♦/ 
/♦  ♦!  -  float  array  pointer  ♦/ 
/♦  -  float  array  pointer  */ 
/♦  *S  -  float  array  pointer  */ 
/*  nodes.in.layer  -  Integer  array  ♦/ 
/♦  starting.node_in_layer  -  Integer  array  */ 
/♦  train.set  -  Integer  ♦/ 
/♦  current.layer  -  Integer  */ 
/♦  */ 
/*  Variables  Roturned:  -  float  array  pointer 

/*  Date;  10  lov  90  Revision:  1.0  */ 


void  calculate.weight. matrix (struct  lode.data  *l.record[3, 

float  ♦W[3, 

float  ♦in, 

float  ♦YH, 
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float  ♦SC] , 

int  nodes.in.layorQ , 

int  start ing.node.in.lay or [] » 

int  pat terns » 

int  current«layer) 

{ 

float  stun  ~  0; 

int  rov,  coition,?,  L,  current .node ,  previous.layer.node; 
for  (row  B  0;  row  <  nodes.in.layer [current . layer-l] ;  row++) 

{ 

preTious.layer.node  »  starting.node.in.layer [current. layer-l]  +  row; 
for  (column  «  0;  column  <  nodes.in.layer  [current. layer] ;  coltimn++) 

{ 

•((W[row])+coltimn)  «  0; 

for  (L  =  0;  L  <  node8.in.layer [current .layer-1] ;  L++) 

{ 

sum  =  0; 

for  (P  =  0;  P  <  patterns;  P++) 

sun  »=  sun  +  ♦((¥[?])+!)  ♦  ♦((S[P])+coltimn) ; 

♦((W[row])+column)  =  *((W[row])+column)  +  ♦((H[row])+L)  ♦  sun; 

} 

current  .node  =  st  art  ing.node.in.layer  [current  .layer]  +  column; 
l_record[current.node]->weight[previous.layer.nod0]  =  ♦((W[row])+column) ; 

} 

> 

> 

/i^t*t*****^****************'y$ir¥*****4*******t*********^***/ 

/♦♦  End  Functions  Called  by  Optimize  Weights  by  Matrix  ♦♦♦/ 


/*  Function  lame:  MSE.last.layer  lumber:  8.5  */ 
/*  Description:  The  function  determines  wether  to  backpropate  the  */ 
/*  parameter  by  the  sigmoidal  or  linear  update  */ 
/*  equations  ♦/ 
/♦  ♦/ 
/*  Functions  Called:  8.6  MSE.last.layer  .linear  ♦/ 
/♦  8.7  MSE.last.layer.sigmoid  ♦/ 
/♦  */ 
/♦  Variables  Passed  In:  lode.record  -  Structure  array  */ 
/*  desired.output  -  Float  array  ♦/ 
/*  nodes.in.layer  -  Integer  array  ♦/ 
/♦  starting_node.in_layer  -  Integer  array  ♦/ 
/*  last.layer  -  Integer  */ 
/*  MSE.eta  -  Float  ♦/ 
/♦  MSE.epsilon  -  Float  ♦/ 
/♦  ♦wght  -  Float  array  pointer  */ 
/♦  MSE.momentem  -  Float  ♦/ 
/♦  */ 
/*  Variables  Returned:  lode.record  -  Structure  array  */ 
/♦  Date:  10  Hov  90  Revision:  1.0  ♦/ 


void  MSE.last.layer (struct  Hode.data  ♦nodo_record[] , 
float  desired.output [] , 
int  nodes.in.layer [] , 
int  starting.node.in.layer[] , 
int  last.layer, 
float  eta, 
float  epsilon, 
float  ♦wght[], 
float  alpha) 

{ 

int  X,  y,  last.layer.node,  previous.layer_node; 
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for  (x  =  0;  X  <  nodes^in.layer Clast «lay or] ;  x++) 

i 

last^layer.node  =  start  ing»node«in..l  ay  er[last«layer]+xj 
snitch  (nodo«re cord [last ^layer«nodo]“>transfer«f unction) 

{ 

case  1: 

HSE.last .lay er^sigmoidCnode .record, 

desired.output , 
nodes.in.layor, 
start ing.node.in.layer, 
last .layer, 
last.layer.node , 
eta, 
wght, 
alpha) ; 

break; 

case  2: 

/♦  reserved  for  rbf  ♦/ 

printf(*'\n  error,  improper  transfer  function'*); 

break; 

case  3: 

HSE.last. lay er.linear (node .record, 

desired.output , 
nodes. in.layer, 

St  art ing.node.in.lay er , 

last. layer, 

last.layer.node , 

eta, 

nght, 

alpha) ; 

break; 
default : 

break; 

} 

} 

} 

/'t  Function  Maine:  MSE.last.layer.linear  Mumber:  8.6  ♦/ 

/*  Description:  This  function  implements  the  following  update  rule  */ 

/*  for  a  node  forming  a  linear  combination  of  the  */ 

/*  outputs  from  the  previous  layer:  ♦/ 

/♦  w+  =  w-  +  eta  ♦  (d-z)  ♦  y  */ 

/♦  ®+  ss  «-  +  eta  ♦  (d-z)  ♦/ 

/♦  where  w+  -  next  weight  */ 

/*  eta  -  training  factor  (.01  -  .99)  ♦/ 

/♦  d  -  desired  output  ♦/ 

/♦  z  -  actual  output  of  the  last  layer  node  ♦/ 

/♦  y  -  actual  output  of  the  node  in  the  previous  layer  ♦/ 

/♦  ®  -  the  sigma  (theta)  for  that  node  ♦/ 

/♦  ♦/ 

/*  Functions  Called:  Hone  ♦/ 

/♦  Variables  Passed  In:  lode.record  -  Structure  Array  */ 

/♦  desired.output  -  Float  array  ♦/ 

/♦  nodes.in.layer  -  Integer  array  ♦/ 

/♦  starting.node.in.layer  -  Integer  array  ♦/ 

/♦  number. of  .layers  -  Integer  ♦/ 

/♦  total.nodes  -  Integer  ♦/ 

/*  MSE.eta  -  Float  ♦/ 

/♦  ♦wght  -  Float  array  pointer  */ 

/♦  MSE.momcntum  -  Float  */ 

/*  ♦/ 

/*  Variables  Returned:  Hodo.record  -  Structure  Array  ♦/ 

/♦  Date:  10  Hov  90  Revision:  1-0  ♦/ 


void  MSE.last^layer^linearCstruct  Kode.data  ♦node.recordC] , 

float  desired.outputC] , 
int  nodos_in«layer[] , 
int  start ing^node.in^layorC] , 
int  last. layer, 
int  last.layer.node, 
float  eta, 
float  ♦wghtCl, 
float  alpha) 

{ 

float  delta.l,  old.nght; 
int  y,  previous.layer.node; 
int  X  =  0; 

X  =  last.layer.node  -  start ing_node_in.layer [last.layer] ; 
delta.l  =  desired.outputCx]  -  node.record[last.layer.node]->output; 
for  (y  =  0;  y  <  nodes.in.layerClast_layer-l] ;  y++) 

{ 

previous.layer.node  =  st art ing.node_in.layer [last. Iayer-l3+y; 
old.wght  =  ♦ ((wght [last.layer .node] )+previous_layer.node) ; 

♦ ( (vght [last.layer.node] )+previous_layer_node)  = 

node.record [last.layer.node] ->weight [previous.layer_node] ; 

node.record[last.layer.node]~>weight [previous.layer.node]  = 
node.record [last.layer.node] -> weight [previous.layer_node] 

+  eta  ♦  delta.l  ♦  nodo_record[previous_layer.node]~>output 
+  alpha  ♦  (node.record [last.layer.node] “>weight[previous.layer.nod0] 

-  old.wght); 

} 

old.wght  =  ♦((wght[last_layer.node])+last_laycr_node) ; 

♦<(wght [last.layer.node] )+last_layer_node)  = 

node.record  [last .layer.node]->signia[last.layer.node] ; 
node.record [last.layer.node] ->sigina[last.layer.node]  = 
node.record[last.layer.node]  •“>sigzna[last.layer.node]  + 
eta  ♦  delta.l 

+  alpha  *  (node.record [last.layer.node]->sigma [last.layer.node] 

-  old.wght); 

} 

/*iti^t***********************^^*********f^****t^**^l^***1tit******nnl^*****^^*/ 

/*  Function  Hame:  MSE.last.layer.sigmoid  Humber:  8.7  */ 

/♦  Description:  This  function  implements  the  update  rule  for  a  ♦/ 

/♦  sigmoidal  transfer  function  in  the  last  layer:  ♦/ 

/♦  w+  =  w-  +  eta  ♦  (d  -  z)  ♦  (1-z)  ♦  z  ♦  y  ♦/ 

/♦  =  C-  +  eta  ♦  (d  -  z)  ♦  (1-z)  *  z  ♦/ 

/♦  whore  ♦/ 

/♦  w  -  weight  between  node  in  last  layer  and  previous  layer  ♦/ 

/*  Q  -  sigma  or  theta  of  last  layer  node  ♦/ 

/♦  eta  “  training  coefficient  ♦/ 

/*  d  -  desired  output  for  the  last  layer  node  */ 

/♦  z  -  actual  output  for  the  last  layer  node  */ 

/♦  y  -  actual  output  for  the  previous  layer  node  ♦/ 

/♦  */ 

/♦  Functions  Called:  Hone  ♦/ 

/♦  Variables  Passed  In:  Hode.record  -  Structure  Array  ♦/ 

/♦  desired.output  -  Float  array  ♦/ 

/*  nodes. iu.layer  -  Integer  array  ♦/ 

/♦  start ing.node.in.layer  -  Integer  array  */ 

/*  current.layer  -  Integer  ♦/ 

/*  currcnt.node  -  Integer  ♦/ 

/*  HSE.eta  -  Float  ♦/ 

/♦  ♦wght  -  Float  array  pointer  ♦/ 

/♦  HSE.momentum  -  Float  ♦/ 
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/♦  ♦/ 
/*  Variables  Returned:  Hode^record  -  Structure  Array  */ 

/*  Date:  10  Hov  90  Revision:  1.0  ♦/ 


void  MSE„last_layer«sigmoid(struct  lode^data  ♦node^recordD , 

float  desired^output[] , 

int  nodes^in.layerD , 

int  starting«node_in_layer[3 , 

int  current.layer, 

int  current ^node, 

float  eta, 

float  *HghtE]> 

float  alpha) 

float  delta^l,  delta«2,  old.wght; 
int  y,  previous^layer.node,  x; 

X  =  current _node  -  starting.node.in.layer [current flayer] ; 
delta.l  =  desired^output[x3-node.record[current_node3**>output; 
d6lta_2  =  (l-nodo_record[current.node]->output) 

*  node_record[current_node3->output; 
for  (y  =  0;  y  <  nodes_in_layer[current.layer“l3 :  y++) 

i 

previous^layer.node  =  starting_node_in«layer[current_layer-l3  +  y; 
old^wght  =  *((wght[current_node])+previous«layer.node) ; 

♦  ( (wght [current ^node] )+previous.3  ayer_node)  = 

node_recordEcurrent_node]->Height[previous»layer^node3 ; 

nod e«re cord  [curr ent .node] -> weight [pro vious«layer_node]  = 

node.record[ctirrent.node]  ->  weight  [previous.laycr.node] 

+  eta  ♦  delta.l  ♦  delta_2  ♦  node.rocord[previous_layor_node]->output 
+  alpha  *  (node.record[currcnt.node]->weight[previous_layer_node] 

-  old.wght); 


> 

old.wght  =  ♦((wght[current.node])+current.nodo); 

♦ ( (wght [current .node] )+current.node)  = 

node.record[current.node]->sigifla[current.node3 ; 
no de.record[current.node]">sigma [current .node]  = 
node.record[current.node]->sigma[current.node]  + 
eta  *  delta.l  ♦  delta.2 

+  alpha  ♦  (node.record[current.node3->sigiua [current .node] 

-  old.wght); 

> 


/♦  Function  Hame:  MSE.inid.layer  Number;  8.8  ♦/ 
/♦  Description;  This  function  implements  update  rule  for  sigmoidal  ♦/ 
/♦  transfer  function  in  the  middle  layer  and  last  layer:  ♦/ 
/♦  y>>+  s  ^}>-  +  eta  ♦  sum(d  -  z)*  w»  ♦  (1-z)  ♦  z  ♦  y  ♦  <l-y)  ♦  x  */ 
/♦  =  Q-  +  eta  ♦  sum(d  -  z)  ♦  w^  *  (1-z)  ♦  z  ♦  y  ♦  (1-y)  */ 
/♦  where  ♦/ 
/♦  w  “  weight  between  node  in  last  layer  and  previous  layer  */ 
/♦  ®  “  sigma  or  theta  of  last  layer  node  ♦/ 
/♦  w*“  weight  linking  node  in  layer  3  to  node  in  layer  2  */ 
/♦  w^^  -  weight  linking  node  in  layer  2  to  node  in  layer  1  */ 
/♦  eta  -  training  coefficient  ♦/ 
/♦  d  -  desired  output  for  the  last  layer  node  ♦/ 
/♦  z  -  actual  output  for  the  last  layer  node  */ 
/»  y  -  actual  output  for  the  layer  2  node  ♦/ 
/♦  X  -  actual  output  for  the  layer  1  node  ♦/ 
/♦  */ 
/♦  Functions  Called:  None  */ 
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/♦  Variables  Passed  Into:  Hode.record  -  Structure  Array  ♦/ 
/♦  desired.output  -  Float  array  ♦/ 
/♦  nodes.in^layer  -  Integer  array  ♦/ 
/*  starting.node.in^layer  -  Integer  array  ♦/ 
/♦  current.layer  -  Integer  ♦/ 
/♦  MSE.eta  -  Float  ♦/ 
/♦  *wght  -  Float  array  pointer  ♦/ 
/♦  MSE^aoncntum  -  Float  ♦/ 
/♦  */ 
/♦  Variables  Returned:  lode.record  -  Structure  Array  ♦/ 
/♦  Date:  10  Hov  90  Revision:  1.0  ♦/ 


void  KSE_mid^layer (struct  lode^data  ♦node.recordQ , 
float  desired [] , 
int  nodes.in^layorD  > 
int  starting^node.in.layerD  > 
int  current flayer, 
float  eta, 
float  ♦ughtC], 
float  alpha) 

int  z,  last. lay er^node,  y,  previous.layer.node,  current .node; 
int  z; 

float  sum,  delta.l,  delta.2,  old.wght; 
sum  =  0; 

for  (x  =  0;  X  <  nodes.in.layer (current .layer] ;  x++) 

{ 

sum  =  0; 

current .node  =  start ing.node.in.laycr [current .layer]  +x; 
for  (z  =  0;  z  <  nodes.in.layer (current _layer+l] ;  z++) 

{ 

last. lay er.node  =  start ing_node.in.layer (current . lay er+1]  +  z; 
delta.l  =  desired(z]  -  node.record(last.layer.node]->output ; 
delta_2  =  (1  -  node.record(last.layer_node]’*>output) 

♦  node.record (last.lay er.node] ->output 

♦  node.record  (last.lay er.node] ->weight (current .node] ; 
sum  5=  sum  +  delta.l  ♦  delta_2; 

} 

delta.l  =  (1  -  node.record (curr ent .node] ->output) 

»  node.record (current.node] ~>output ; 

for  (y  =  0;  y  <  nodes.in.layer (current .lay er-1] ;  y++) 

{ 

previous_layer.node  =  start ing.node.in.layer (curr ent .lay er-1]  +  y; 
old.wght  =  ♦ ( (wght (current .node] )+prcvious_layer.node) ; 

♦ ( (wght (current .node] )+previous.laycr.node)  = 

node.record (current.node] ->weight (previous.layer.node] ; 

node.record (current.node] -> we ight(previous.lay er.node]  = 

node.record (current .node] -> weight [previous.lay er.node] 

+  eta  ♦  delta.l  ♦  sum  ♦  node.record (previou8.1ayer.node]->output 
+  alpha  ♦  (node.record (current.node]->weight (previous.lay er.node] 

-  old.wght); 


old.wght  =  ♦ ( (wght (current .node] ) +current_node) ; 

♦ ( (wght (current .node] )+curr ent .node)  = 

node.record [current.node] -> sigma (current.node] ; 
node.record (current .node] ->sigma(currcnt.nodc]  = 
node.record (current.node] ->sigma (current .node]  + 
eta  *  delta.l  *  sum 
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+  alpha  ♦  (node_recordCcurrent«nodo3-> sigma [current .node] 
-  old.wght); 

} 

> 


/*  Function  lame:  HSE.ist.layer  lumber:  8.9  */ 
/*  Description:  This  fxinction  implements  update  rule  for  sigmoidal  ♦/ 
/*  transfer  function  in  the  firsts  middle  layer  and  last  layer:  */ 
/♦  s  4.  eta*sum(d-z)^(l-z)*z*8um(v«y*(l-y)*v’ ^)«x*(l-x)*a  ♦/ 
/*  C+  =  C-  +  eta  ♦  sumCd  -  z)*(l~z)*z*sum(w*y*(l-y)*w’ ^)*x*(l“x)  ♦/ 
/♦  uhere  ♦/ 
/♦  w  -  veight  between  node  in  last  layer  and  previous  layer  ♦/ 
/*  €  ~  sigma  or  theta  of  last  layer  node  */ 
/*  w>~  veight  linking  node  in  layer  3  to  node  in  layer  2  ♦/ 
/*  -  veight  linking  node  in  layer  2  to  node  in  layer  1  ♦/ 
/♦  v>>>  -  veight  linking  node  in  layer  1  to  node  in  layer  0  */ 
/*  eta  -  training  coefficient  ♦/ 
/♦  d  -  desired  output  for  the  last  layer  node  ♦/ 
/♦  z  -  actual  output  for  the  last  layer  node  ♦/ 
/♦  y  “  actual  output  for  the  layer  2  node  ♦/ 
/♦  X  “  actual  output  for  the  layer  1  node  ♦/ 
/♦  a  ~  actual  output  for  the  layer  0  node  ♦/ 
/*  ♦/ 
/♦  Variables  Passed  into:  Node.record  -  Structure  Array  ♦/ 
/♦  desired.output  -  Float  array  ♦/ 
/♦  nodcs.in.laycr  -  Integer  array  ♦/ 
/♦  starting.node.in_layer  -  Integer  array  ♦/ 
/♦  current.layer  -  Integer  ♦/ 
/♦  MSE.eta  -  Float  ♦/ 
/♦  *vght  -  Float  array  pointer  ♦/ 
/♦  MSE.momentum  -  Float  ♦/ 
/*  ♦/ 
/♦  Variables  Returned:  Hode.record  -  Structure  Array  ♦/ 
/♦  Date:  10  Hov  90  Revision:  1.0  ♦/ 


void  MSE.lst.layor(struct  Hode.data  ♦node.recordQ , 

float  desired.outputD , 

int  nodes_in.layer[] , 

int  start ing.node.in.layor  []  , 

int  current .layer, 

float  eta, 

float  ♦wght [] , 

float  alpha) 

{ 

int  next.layer.node,  last.laycr.node,  y,  z,  previous.layer.node; 
int  current .node,  x; 

float  sum.l,  8um.2,  delta.l,  delta.2,  dclta.3,  delta.4; 
float  old.vght; 
sum.l  =  0; 

for  (x  =  0;  X  <  nodcs.in.layer [current. lay er] ;  x++) 

{ 

sum.l  =  0; 

current .node  =  starting.nodc.in.layer[currcnt.layer]  +  x; 
for  (  z  =  0;  z  <  nodcs.in.layer  [current. lay  er+2]  ;  z++) 

last.layer.node  =  starting.node.in.layer [current. lay er+23  +  z; 

delta.l  =  desired.output [z3“node.rccord[last.laycr.node3->output ; 

delta.2  =  (1-nodc.rccord [last. lay er.node3->output) 

♦  node.record [last.layer.node3 ->output ; 

sum.2  =  0; 

for  (y  =  0;  y  <  nodcs.in.layer  [current  .layer  +  13;  y++) 

i 
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next.layer.node  =  starting.node.in.layer [current. layer  +  1]  +  y; 
delta.3  =  (l~node_recordEnext.layer.node3->output) 

♦  node.record  [next.layer.node] ->output ; 

delta.4  =  node.r ccord [last .layer.node]->weight [next.layer.node] 

♦  node.record  [next.layer.node]  “>weight  [current  .node] ; 
sua.2  =  sum_2  +  delta.3  ♦  delta.4; 

} 

sum.l  =  snin.l  +  delta.l  *  delta.2  ♦  sum_2; 

} 

delta.l  =  (1  -  node.record  [current .node]  “>output) 

*  node.record  [current .node]  ~>output ; 
for  (y  =  0;  y  <  nodes.in.layer[current.layer~l] ;  y++) 

{ 

previous.layer.node  =  starting.node.in.layer [current .layer-1]  +  y; 
old.ught  =  ♦ ( (vght [current .node] )+preTious.layer.node) ; 

♦ ( (vght [current .node] )+previous.layer.node)  = 

node.record [current.node]->veight [provious.layer.node] ; 
node.record [current.node]-> weight [previous.layer.node]  = 
node.record[currcnt.node]->weight [previous.layer.node] 

+  eta  ♦  delta.l  ♦  sum.l  ♦  node.record [previous.layer.nodo]->output 
+  alpha  ♦  (node.record[current.node]->weight [previous.layer.node] 

-  old.wght); 

} 

old.wght  =  ♦( (wght[current. node] )+current. node ) ; 

♦ ( (wght [current .node] )+current.node)  = 

node.record [current.node]->sigma[current.node] ; 

node.record [current .nodeJ->sigma[current .node]  = 
node.record[current .node] ->sigma[current .node]  + 
eta  ♦  delta.l  ♦  sum.l 

+  alpha  ♦  (node.record[current.node]->sigma[current.node] 

-  old.wght); 

} 

} 

/**^**i¥**t4****t*****^t*****^0*i^t*****tt*ttt*t****************0t*****/ 


/*  Function  lameicalculate.errors.in.output  dumber:  8.10  ♦/ 
/*  Description:  This  function  compares  the  output  of  each  node  in  */ 
/*  in  the  output  layer  with  the  desired  output.  ♦/ 
/*  If  the  difference  between  the  desired  and  the  ♦/ 
/*  actual  is  greater  than  some  delta,  an  error  ♦/ 
/♦  is  returned  ♦/ 
/♦  Functions  Called:  Bone  ♦/ 
/♦  Variables  Passed  In:  lode.record  -  Structure  Array  ♦/ 
/♦  desircd.output  ~  Float  array  */ 
/♦  nodes.in.layer  -  Integer  array  ♦/ 
/♦  starting.node.in.layer  -  Integer  array  */ 
/♦  number,  of  .layers  -  Integer  */ 
/*  ♦error  -  Integer  pointer  ♦/ 
/♦  epsilon  -  Float  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  ♦error  -  Integer  pointer  ♦/ 
/♦  Date:  10  Bov  90  Revision:  1.0  ♦/ 


/:^f**t****t**t********^^*********t***^***t******i¥^****t*******0******/ 

void  calculate.errors.in.output (struct  Bode.data  ♦node.record [] , 

float  desired.output [] , 
int  nodes.in.layer[] , 
int  starting.node.in.layer[] , 
int  number.of. layers, 
int  ♦error, 
float  epsilon) 

int  X,  last.layer.node; 

for  (x  =  0;  X  <  nodes.in.layer [number.of. layers] ;  x++) 
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{ 

last«layer_node  =  start ing_node«in.layerCnumber.of«layers]+x; 
if  (fabs(desircd_output  W-node«record [last .lay cr«node3~>output)  >  epsilon) 
{ 

♦error  “  ♦error  +  1; 

X  =  nodes.in.layer[number.of. layers]  ; 

} 

} 

> 

/♦♦♦♦♦  End  functions  called  by  backprp  remaining  lyrs  ♦♦♦♦♦/ 


/000000000:$000^^0000001^0000^000000000^0000^00m0*0000*000000/ 

/♦  The  following  functions  called  by  train  via  kohonen  ♦/ 
/000000000000000^0000000000^0000000000000000000000000000000/ 

/000000000000000i$0000000000t^0000000000000000000000000^00000000000000/ 

/♦  Function  Vane:  get.linear.training.eta  Humber:  8.11  ♦/ 

/♦  Description:  This  function  determinse  the  training  eta  by  ♦/ 

/♦  n  =  {n(max)/[i(o)~i(max)]}[i-i(o)]+n(inax)  ♦/ 

/♦  Functions  Called:  Hone  ♦/ 

/♦  Variables  Passed  In:  train.vidth  -  Integer  array  ♦/ 

/♦  train.scale  -  Float  array  ♦/ 

/♦  iterations  ~  Integer  ♦/ 

/♦  ♦eta  “  Float  pointer  ♦/ 

/♦  vidth.no  -  Integer  ♦/ 

/♦  ♦/ 

/♦  Variables  Returned:  ♦eta  -  Float  pointer  ♦/ 

/♦  Date:  10  Vov  90  Revision:  1.0  ♦/ 


/♦♦♦♦♦♦♦♦♦♦♦*♦♦♦«♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦«♦*♦♦««♦*«♦♦♦«♦♦♦♦♦*♦♦♦«♦*♦♦/ 

void  get.linear.training.eta  (int  train.vidth [] , 

float  train.scale [] , 
int  iterations, 
float  ♦eta, 
int  vidth.no) 

int  n,  x; 

n  =  0; 

for  (x  »  1;  X  <  vidth.no;  x++) 
if  (iterations  >  train.vidth[x]) 
n  =  x; 

♦eta  =  (train.scale [n]/ (train.vidth [n]  -  train.vidth [n+l])) 

♦  (iterations  -  train.vidth [n])  +  train.scale [n] ; 

> 

/^^(♦♦♦♦♦♦♦♦^♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦^ ♦♦♦♦♦♦♦♦♦*♦♦♦♦♦/ 


/♦  Function  Vane:  get.kohonen.neighborhood  Humber:  8.12  ♦/ 
/♦  Description:  This  function  provide  the  neighborhood  used  in  */ 
/♦  the  update  of  the  nodes  ♦/ 
/*  */ 
/♦  Functions  Called;  Hone  ♦/ 
/♦  Variables  Passed  In;  train.vidth  -  Integer  array  ♦/ 
/♦  iterations  -  Integer  ♦/ 
/♦  neighborhoods  -  Integer  array  ♦/ 
/♦  vidth.no  -  Integer  ♦/ 
/♦  ♦neighbor  -  Integer  pointer  ♦/ 
/♦  ♦/ 
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/♦  Variables  Returned:  ^neighbor  -  Integer  pointer  */ 

/*  Date:  10  lov  90  Revision:  1.0  ♦/ 

voi<*  get.kohonen^neighborhood  (int  train.vidthC] , 

int  iterations* 
int  neighborhoodsC]  * 
int  width.no* 
int  ^neighbor) 

{ 

int  n*  x; 

n  =  0; 

for  (x  =  1;  X  <  width.no;  x++) 
if  (iterations  >  train.widthCx]) 
n  =  x; 

♦neighbor  =  neighborhoods [n] ; 

} 


/♦  Function  lane:  calc.dist.outputs.to.nxt.lyr  Humber: 8. 13  ♦/ 
/*  Description:  This  function  finds  the  euclidean  distance  between  */ 
/*  the  outputs  of  one  layer  and  the  weights  of  the  ♦/ 
/♦  next  layer  ♦/ 
/*  d(ij)  =  sqrt{sum[y(i)-w(j)]"2}  ♦/ 
/♦  Functions  Called:  Hone  ♦/ 
/♦  Variables  Passed  In:  lode.record  -  Structure  array  ♦/ 
/♦  nodes.in.layer  -  Integer  array  ♦/ 
/♦  starting.node.in.layer  -  Integer  array  ♦/ 
/♦  current.layer  -  Integer  */ 
/♦  distance  -  Float  array  */ 
/♦  */ 
/♦  Variables  Returned:  distance  ~  Float  array  */ 
/♦  Date:  10  lov  90  Revision:  1.0  ♦/ 


void  calc.dist.outputs.to.nxt.lyr (struct  lode.data  ♦node.recordC] > 

int  nodes.in.layer [] * 
int  start ing.node.in.layer [] * 
int  current.layer* 
float  distanced) 


{ 

int  X,  y*  current .node*  previous.layer.node; 
double  buffer; 
double  exponent.l  =  2; 

for  (x  =  0;  X  <  nodes. in.layer [current . layer] ;  x++) 

current .node  =  starting.node.in.layer [current.layer]  +  x; 
buffer  =  0; 
distance  [x]  =  0; 

for  (  y  =  0;  y  <  nodes.in.layer [current .lay er-1] ;  y++) 

-C 

previous.layer.node  =  starting_node.in.layer[current.layer-l]  +  y 
buffer  =  node.record [current . node] “>weight[previous.layer.node] 

-  node.record [previous.lay er.node] ->output ; 
distance [x]  =  distance [x]  +  pow (buffer* exponent.l); 

} 

} 


/i^t***************i¥**i^******f^**0***t**0****^*****0*f¥******t^*********/ 

/♦  Function  Hame:  f ind_n6arest.elem6nt  Humber:  8.14  */ 
/*  Description:  This  function  finds  the  node  in  the  kohonen  layer  */ 
/*  nearest  to  the  input  pattern  ♦/ 
/♦  Functions  Called:  Hone  ♦/ 
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/♦  Variables  Passed  In:  min  -  Float  array  ♦/ 
/♦  array^max  -  Integer  ♦/ 
/*  ♦nearest.element  -  Integer  pointer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  ♦nearest.element  -  Integer  pointer  ♦/ 
/*  */ 
/♦  Date:  10  lov  90  Revision:  1.0  ♦/ 


void  find.nearest.element (float  mln[], 

int  array.max> 

int  ♦nearest.element) 


{ 

int  x; 

float  temp  =  1000; 
♦nearest^element  =  0; 
for  (x  =  0;  X  <  array .max;  x++) 
if  (temp  >  fflinCx]) 

{ 

temp  =  min[x]; 
vnearest.element  =  x; 

} 

> 


/*  Function  lame:  find.kononen.boundaries  lumber:  8.15  */ 
/*  Description:  This  function  finds  the  valid  boundaries  of  the  */ 
/*  rectangular  kohonen  layer.  Theses  boundaries  are  */ 
/♦  centered  at  the  winning  node  in  the  network  ♦/ 
/♦  ♦/ 
/♦  Functions  Called:  lone  ♦/ 
/♦  Variables  Passed  In:  winner.node  -  Integer  ♦/ 
/*  starting.node.in_layer  -  Integer  array  ♦/ 
/♦  current .layer  -  Integer  */ 
/♦  nodes.x  -  Integer  */ 
/♦  nodes.y  -  Integer  ♦/ 
/♦  neighbors  -  Integer  ♦/ 
/♦  eboundary.left  -  Integer  pointer  ♦/ 
/♦  eboundary.right  -  Integer  pointer  ♦/ 
/♦  ♦boundary.up  -  Integer  pointer  ♦/ 
/♦  ♦boundary .down  -  Integer  pointer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  ♦boundary .left  -  Integer  pointer  ♦/ 
/♦  ♦boundary.right  -  Integer  pointer  ♦/ 
/♦  ♦boundary. up  -  Integer  pointer  ♦/ 
/♦  ♦boundary.down  -  Integer  pointer  ♦/ 
/♦  ♦/ 
/♦  Date:  10  lov  90  Revision:  1.0  ♦/ 


void  f ind.kohonon.boundaries  (int 

int 

int 

int 

int 

int 

int 

int 

int 

int 

{ 


winner. node, 

start ing.node.in.layerC] , 

current .layer, 

nodes.x, 

nodes.y, 

neighbors, 

♦boundary.lcf t , 
♦boundary.right , 
♦boundary. up, 
♦boundary.down) 


int  neighbors.x,  neighbors.y,  winner .node.y,  w inner. node.x; 
neighbors.x  =  neighbors  -  1; 
neighbors.y  =  neighbors  -  1; 
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wiimer^node.y  »  (winner_node“Start2iig_iiode^in«lay9r [current .layer]) 
/nodes.x; 

vinner.nodo.x  »  (winner.node~8tarting.node.in.layer[current_layer]) 
,  -  winner.node.y  ♦  nodes.x; 

♦boundary.loft  “  winner.node.x  -  neighbors.x/2 1 
♦boundary. right  =  vinner.node.x  +  neighbors.x/2; 

♦boundary  .up  =  winner.node.y  +  neighbors.y/2; 

♦boundary. do «n  =  ainner.node.y  -  neighbors.y/2; 

if  (♦boundary.left  <  0) 

♦botindary.left  =  0; 
if  (♦boundary.right  >  nodes.x  -  1) 

♦boundary.right  =  nodes.x  -  1; 
if  (♦boundary. up  >  nodes.y  ~  1) 

♦boundary .up  =  nodes.y  -  i; 
if  (♦boundary. down  <  0) 

♦bound iry. down  =  0; 


} 


/♦  Function  lame:  determine.neighborhood.elcments  Humber:  8.16  ♦/ 


/♦  Description:  This  function  returns  the  node  ntimbers  of  the  ♦/ 
/♦  nodes  whose  weights  will  bo  updated  ♦/ 

/*  ♦/ 

/♦  Functions  Called:  Hone  */ 

/♦  Variables  Passed  In:  boundary.left  -  Integer  ♦/ 

/♦  boundary.right  ~  Integer  ♦/ 

/♦  boundary.up  -  Integer  ♦/ 

/♦  boundary.down  -  Integer  ♦/ 

/♦  ♦node s.t ©.update  -  Integer  pointer  ♦/ 

/♦  update.node  -  Integer  array  ♦/ 

/♦  starting.node.in.layer  -  Integer  array  ♦/ 

/♦  nodes.x  -  Integer  ♦/ 

/♦  current.layer  -  Integer  ♦/ 

/»  */ 

/♦  Variables  Returned:  ♦node s.to.upd ate  ~  Integer  pointer  ♦/ 

/♦  update.node  -  Integer  array  ♦/ 

/♦  Date:  10  Hov  90  Revision:  1.0  ♦/ 


void  determine.neighborhood. elements  (int  boundary.left, 

int  boundary.right, 

int  boundary.up, 

int  boundary.down, 

int  ♦nodes.to.update, 

int  update.node[] , 

int  starting.node_in.layer[] , 

int  nodes.x, 

int  current.layer) 

{ 

int  X,  y,  z; 

2  =  0; 

for  (y  =  boundary.down;  y  <  bovT)dary.up  +  1;  y++) 
for  (x  =  boundary.left;  x  <  oiidary.right  +  1;  x++) 

e 

X 

update.node [z]  =  x  +  nodes.x  ♦  J  + 
start ing.nodo.in.layer [current.layer] ; 

2  =  2  +  1; 


} 

♦nodes.to.update  =  z; 
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/*  Function  Inmo:  train^kohonen^veights  lumber:  8.17  */ 
/*  Description:  This  function  updates  the  vinning  node  and  the  */ 
/*  neighborhood  nodes  by  the  equation  */ 
/*  w(+)  =  w(“)  +  nCx-u(-)]  ♦/ 
/*  */ 
/*  Functions  Called:  lone  */ 
/*  Variables  Passed  In:  lode.record  -  Structure  array  ♦/ 
/♦  nodes. in.layer  -  Integer  array  */ 
/*  starting.node.in.layer  -  Integer  array  ♦/ 
/♦  current.layer  -  Integer  array  ♦/ 
/♦  nodes.to.update  -  Integer  ♦/ 
/♦  update.node  -  Int<>ger  array  */ 
/*  eta  -  Float  ♦/ 
/*  ♦/ 
/♦  Variables  Returned:  lode.record  -  Structure  array  ♦/ 
/*  Date:  10  Hov  90  Revision:  1.0  ♦/ 


/^t*************************************************tt****^*0**j^4*^*/ 

void  train.kohonen.weights  (struct  lode.data  *node.record[3 , 

int  nodes.in.layerD , 

int  start ing.node.in.layer [] , 

int  current .layer, 

int  nodes.to.update, 

int  update.nodeC] , 

float  eta) 

{ 

int  X,  y,  current .node,  previous.layer.node; 
float  buffer; 

for  (x  =  0;  X  <  nodes.to.update;  x++) 

{ 

current .node  =  update.node[x] ; 

for  (y  =  0;  y  <  nodes.in.lay er [current .layor-1];  y++) 

{ 

previous.layer.node  =  starting.node.in.layer[current.layer-l]  +  y; 
buffer  =  node.recordCprevious.layer.node]->output  ~ 

node .record [current .node] -> weight [previous.layer.node] ; 
node.record[current.node] ->weight [previous.layer.node]  = 

node.re cord  [current .node]  ->wei^t  [previous.layer.node] 

+  eta  ♦  buffer; 

> 

} 

} 

/♦  End  functions  called  by  train  vi  kohonen  */ 

/i^i^^0t************************i^*0***j^****i^**nf************/ 

/^,lf4;t***********i¥*i^*****************************J^********t’¥/ 

/♦  The  following  functions  are  called  by  set  sigma  at  */ 

/♦  P  neighbors  avg.  ♦/ 


/*  Function  Same:  find.distance.between.nodes  lumber :8. 18  */ 

/*  Description:  This  function  finds  the  euclidean  distance  between  */ 


/*  nodes  in  the  same  layer  */ 
/*  d(ij)  =  sqrt{8um[w(i)-w(j)]"2}  ♦/ 
/*  Functions  Called:  lone  ♦/ 
/♦  Variables  Passed  In:  lode.record  -  Structure  array  ♦/ 
/*  nodes. in.layer  -  Integer  array  ♦/ 
/*  starting.node.in.layer  -  Integer  array  ♦/ 
/♦  current.node  -  Integer  ♦/ 
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/♦  next.node  -  Integer  ♦/ 
/♦  current«layer  -  Integer  ♦/ 
/*  ♦distance.between  -  Float  array  pointer  */ 
/♦  ♦/ 
/*  Variables  Returned:  >»distance.b6t«een  -  pointer  «/ 
/*  Date:  10  lov  90  Revision:  1.0  ♦/ 


void  find.distance^between.nodes (struct  Vode.data  ♦iiode.recordQ  , 

int  nodes.in^layerD , 

int  starting^node^in.layerD , 

int  current .node, 

int  next.node, 

int  current .layer, 

float  ♦distance.betveenC]) 

{ 

int  X,  y,  z,  previous.layer.node; 
double  distance,  buffer; 
double  exponent.l  =  2; 
double  exponent.2  =  .5; 

X  =  current .node  -  start ing.node.in.layer [current .layer] ; 
y  =  next.node  -  starting.node_in.layer [current .layer] ; 
buffer  =  0; 

for  <2=0;  z  <  nodes. in.layer [current .layer-1] ;  z++) 

{ 

previous.layer.node  =  start ing_node.in.layer[current.laycr-l]  +  z 
distance  =  node.rocord [current. node] ->Height[previous_layer.node] 
-  nodo.record [next.node] ->weight[previous.layer.node] ; 
distance  ~  pou(distance,exponent.l); 
buffer  =  buffer  +  distance; 

} 

♦ ((distance.be tween [x])+y)  =  pow (buffer, exponent. 2) ; 


/♦  Function  lane:  sort.2.dim. array  Rumber:  8.19  ♦/ 

/«  Description:  This  function  returns  an  array  sorted  in  ♦/ 

/♦  descending  order  ♦/ 

/*  ♦/ 

I*  Functions  Called:  lone  */ 

/♦  Variables  Passed  In:  ♦M  -  Float  array  pointer  ♦/ 

/♦  array .max  -  Integer  */ 

/♦  row  -  Integer  ♦/ 

/♦  ♦/ 

/♦  Variables  Returned:  ♦M  -  Float  array  pointer  */ 

/♦  Date:  10  Hov  90  Revision:  1.0  ♦/ 


void  s or t.2_diiD_array (float  ♦!![], 

int  array .max, 
int  row) 

int  y,  z; 
float  temp; 

for  (y  =  0;  y  <  array .max;  y++) 
for  (z  =  y;  z  <  array_max;  2++) 
if  (*((K[ro!?])*2)  <  *((!![ro5]>*y)  ) 

temp  =  ♦((M[row])+y) ; 
*((M[row])+y)  =  ♦((M[row])+z) ; 
♦((M[row])+2)  =  temp; 

} 

} 
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/*  End  functions  called  by  set  sigma  at  P  neighbor  avg  **/ 


/♦  Functions  Called  by  CE  Remaining  Layers  */ 

/*  Function  lame:  CE.last .layer  lumber: 8. 20  */ 
/*  Description:  This  function  updates  the  last  layer  weights  of  e/ 
/*  and  offsets  of  a  sigmoidal  network  by  e/ 
/♦  w(mn)+  =  w(mn)-  +  n/(2.31)  Cd(n)“y(n)]y(m)  ♦/ 
/♦  C(n)  =  C(n)  -  n/(2.3I) Cd(n)-y(n)3  ♦/ 
/♦  */ 
/«  Functions  Called:  lone  «/ 
/♦  Variables  Passed  In:  lode.record  -  Structure  array  */ 
/♦  nodes.in.layer  -  Integer  array  ♦/ 
/*  start ing.node.in.layer  “  Integer  array  ♦/ 
/♦  current.layer  -  Integer  ♦/ 
/♦  ♦wght  -  Float  array  ♦/ 
/♦  CE.eta  -  Float  ♦/ 
/♦  CE.momentum  -  Float  */ 
/♦  desired  -  Float  array  */ 
/*  */ 
/*  Variables  Returned:  Vode.record  -  Structure  array  ♦/ 
/♦  Date:  10  Iot  90  Revision:  1.0  ♦/ 


void  CE.last.layor (struct  lode.data  ♦node.recordD , 
int  nodes.in.layerD , 
int  starting.node.in.layerD , 
int  current .layer, 
float  ♦wght  n , 
float  eta, 
float  alpha, 
float  desired []) 

{ 

int  X,  y,  last.lyr.node ,  mid.lyr.node ; 
float  old.wght,  buffcr.l; 

for  (x  =  0;  X  <  nodes.in.layer [current .lay er] ;  x++) 

{ 

last.lyr^node  ^  start ing.node.in.layer [current .lay er]+x; 
buffer..!  dosired[x]  -  node.rccord[last.lyr.node]“>output; 

{ 

fox  (y  =  0;  y  <  nodes.in.layer [current. layer-1] ;  y++) 

{ 

niid.lyr_node  =  starting.node.an.laycr [current .layer-1]  +  y; 
old.wght  »  ♦ ( (wght [last_lyr.node] )+mid.lyr.node) ; 

< ( (wght ilast.lyr.node] )*haid.lyr.nodc)  = 

node.record [last _lyr.node]-> weight  Dnid.lyr.nodc] ; 
nod  a.record[last.lyr .node] ->weight  Daid.lyr.no de]  += 

eta  *  buffer.l  *  node.record [mid.lyr.node]-> out put 
+  alpha 

♦  (node.record [last .lyr. node] ->weight  Dnid.lyr.node] 
-  old.wght) ; 

} 

old.wght  =  ♦ ( (wght (last.lyr.node] )+last.lyr.node) ; 

(ught [las t. lyr . node] )+last.lyr.nodc)  = 

nodc.r e cord [last .lyr. node] -> sigma [las t.lyrjnode] ; 
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node.TQCord [last.lyr_node3 “>sigma [last^lyr^nodo]  += 
eta  *  buffer.l 
+  alpha 

♦  (node.record  [last  .lyr^node]  “>sigina  [last  «lyr  .node] 
-  old.wght); 

} 

} 

} 


/*  Function  Name:  CB.mid.layer  lumber:  8.21  ♦/ 
/*  Description:  This  function  updates  the  weights  in  the  middle  ♦/ 
/♦  layer  of  a  sigmoidal  network  by  ♦/ 
/♦  w(LM)+  =  w(LH)-  +  n/(2.3H)sumCd(n)“y(n)3w(Mn)y(M)  Cl-y(M)]y(L)  ♦/ 
/*  «(LM)+  =  «(LM)“  +  n/(2,3S)sum[d(n)"y<n)3w(Kn)y(M)  Cl~y<M)3  */ 
/*  */ 
/♦  Functions  Called:  Hone  ♦/ 
/♦  Variables  Passed  In:  Hodo.record  -  Structure  array  ♦/ 
/♦  nodes.in.layer  -  Integer  array  ♦/ 
/*  starting.node.in.layer  -  Integer  array  ♦/ 
/♦  current.layer  -  Integer  ♦/ 
/♦  *wght  -  Float  array  ♦/ 
/♦  CE.eta  -  Float  ♦/ 
/*  CE.comentum  ~  Float  ♦/ 
/*  desired  -  Float  array  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  Hone  v/ 
/♦  Date:  10  Hov  90  Revision:  1.0  ♦/ 


void  CE.mid.layer (struct  Bode.data  ^nodo.recordC] , 
int  nodes.in_layerC3 , 
int  start ing.node_in_layer C3 , 
int  current. layer, 
float  ♦wght [3 , 
float  eta, 
float  alpha, 
float  desired [3 ) 

i 

int  X,  y,  z,  mid.lyr.node ,  first.lyr.ncde,  last.lyr.node; 

float  buffer.l,  buffer.2,  old.wght; 

for  (x  =  0;  X  <  nodes.in.layer [current .layer] ;  x++) 

{ 

mid.lyr.node  ~  start ing.node.in.lay erCcurrent .layer] +x; 
buffer.l  =  (l-node.record[mid_lyr.nodc]~>output) 

♦  node.record [mid.lyr.node] ->output ; 
buffer.2  =  0; 

for  (y  =  0;  y  <  nodes.in.layer [current. lay er+l] ;  y++) 

last.lyr.node  =  start  ing.node.in.layer  [current. layer+1]  +  y; 
buffer.2  +=  (desired  [y] -node.record  [last.lyr.node] ‘•>output) 

♦  node.record  [last.lyr.node] ->weight  [mid.lyr.node]  ; 

} 

for  (z  =  0;  z  <  nodes_in.laycr[current.layer-l] ;  z++) 

{ 

first. lyr.node  -  starting.node.in.layer[current.layer-l]  +  z; 
old.wght  =  ♦( (wght rmid.lyr.node])+first.lyr .node) ; 

*  (  (i?ght  [mid.lyr.node]  )■*'!  irst.lyr.node)  = 

node.record [mid.lyr.node] ->weight[first.lyr .node] ; 
node.record [mid.lyr.node]-> weight [first .lyr.node]  += 

eta  *  buffer.2  ♦  buffer.l  ♦  node.r ccord [first .lyr.node] ->output 
+  alpha 

*  (nodo.rccord[mid.lyr.node]->wcight[first.lyr.node]-  old.wght) ; 

} 
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old.wght  =  ♦ ( (wght  Cmid_lyr.node] )+inid«lyr«node) ; 

♦  (  (wght  Cmid_lyr_iiode]  )+mid_lyr^node)  = 

node  _rocordDaid«lyr_nod0]->sigmaDnid_lyr  .node] ; 
node.recordCmid_lyr.node] ->sigmR|jnid.lyr  .node] += 
eta  *  buffer.2  ♦  buifer.l 
+  alpha 

♦  (node.recordCmid.lyr_node]~>sigmaDnid.lyr.node]  -  old.iight); 

} 

} 


I*  Function  lame:  CE.first. layer  lumber:  8.22  ♦/ 
/«  Description:  This  function  updates  the  weights  in  the  first  «/ 
/♦  layer  of  a  three  layer  sigmoidal  network  by  ♦/ 
/*  w(RL)+  =  w(KL)+n/(2.3H)sumCd(n)-y(n)]{sum[w(mn)y(m)(l“y(m))  ♦/ 
/♦  w(Lm)y(L)(l-y(L)y(K)]  ♦/ 
/♦  w(KL)+  =  w(KL)+n/(2.3R)sum[d(n)-y(n)]{sum[w(mn)y(m)(l“y(m))  */ 
/*  w(Lm)y(L)(l-y(L)y(K)]  ♦/ 
/♦  */ 
/*  Functions  Called:  lone  ♦/ 
/♦  Variables  Passed  In:  lode.record  -  Structure  array  ♦/ 
/♦  nodes.in.layer  -  Integer  array  ♦/ 
/♦  otarting.node_in.layer  -  Integer  array  ♦/ 
/♦  current .layer  -  Integer  ♦/ 
/*  *wght  -  Float  array  ♦/ 
/*  CE.eta  -  Float  ♦/ 
/♦  CE.momentum  -  Float  ♦/ 
/♦  desired  -  Float  array  ♦/ 
/♦  */ 
/♦  Variables  Returned:  Node.record  -  Structure  array  ♦/ 
/♦  Date:  10  iov  90  Revision:  1.0  ♦/ 


void  CE.first.laycr  (struct  lode.data  ♦node.recordC]  , 
int  nodes.in.layer [] , 
int  starting.node.in_layer[] , 
int  current .layer, 
float  ♦wght [] , 
float  eta, 
float  alpha, 
float  dosiredC], 
int  total.nodes) 

{ 

float  old.wght,  buffer.l,  buffer.2,  output [TOTAL.SODES] ; 

int  X,  y,  2,  lost. lyr .node,  mid.lyr.node ,  first.lyr_node,  input .lyr .node; 

for  (;:  =  0;  x  <  total.nodes;  x++) 

output [x]  «  (l-node.recordCx]->output) 

*  node.record Cx]->output; 

for  (x  =  0;  X  <  nodes.in.layer  [current  .lay  er] ;  x++) 

{ 

first,  lyr .node  =  starcing_node_in.laycrCcurrent_layer]  +  x; 
buife*-.l  =  0; 

for  (y  =  0;  y  <  nodes.in.lay-'  'rent .lay er+2] ;  y++) 

lsist.lyr.nod«  =  starting.node.7ji_layer  [cur  cent. layer +2]  +  y; 
buffer.2  =  0; 

for  (z  =  0;  2  <  nodes.in.layer [current .lay er+l] ;  z++) 

mid.lyr.node  =  starting.node.in.layor [current _laj  .+l]+z; 
buffer.2  +=  node.record [last. lyr.node]->oeight[mid_lyr_node] 

*  output [mid.lyr.node] 

♦  node.record [mid_lyr.node] ->ueight [first. lyr  jaode] ; 
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> 

buffer.l  +=  buffer^2  ♦  (desire d[y]“node«record [last.lyr.node] ->outp\it ) ; 

} 

for  (y  =  0;  y  <  nodes»in«layerCcurrexit«layer-l] ;  y++) 

input .lyr.node  =  starting«node«in«layerCcurrent.layer-l]+y; 
old^wght  =  ♦ ((wght [first. lyr.node3)+input_lyr .node) ; 

♦ ( (Rght [f irst.lyr.node] )+input.lyr.node)  = 

node.record [first. lyr.node] ~>Beight [input .lyr. node] ; 
node.record[f irst.lyr.node] ->weight [input .lyr .node]  += 
eta  ♦  buffer.l  ♦  output [first. lyr.node] 

♦  node.record[input.lyr.node] ->output 
+  alpha 

♦  (node.record [first.lyr.node] ->weight [input .lyr.node] 

-  old.wght) ; 

} 

old.wght  =  ♦ ( (vght [f irst.lyr.node] )+f irst.lyr.nodo) ; 

♦ ( (vght [f irst.lyr.no de] )+f irst.lyr.node)  = 

node.record[f irst.lyr.node] ->sigma[f irst.lyr.node] ; 

node.record[first.lyr.node]~>sigma[f irst.lyr.node]  += 
eta  ♦  buffer.l  *  output [f irst.lyr.node] 

+  alpha  ♦  (node.r ecord[f irst.lyr.node] ->sigma[f irst.lyr.node] 

-  old.vght); 

> 

> 

/**^*t*******nt**^*********4*t********^********^f*t******t**/ 

/*  End  Functions  Called  by  CE  Remaining  Lyrs  */ 

/**^^^Hi^i^t*^****tt******************t***************i^***i$*/ 

/*  Functions  called  by  CFM  Remaining  Lyrs  ♦/ 

/*^j(^t*********^^t**********i^*t**************************t*/ 

/i^^^^*m**^**i¥****************0************^*^*t*t^********^****i^*****/ 

/*  Function  Fame:  calculate.zn  Bumber:  8.23  */ 

/*  Description:  This  function  calculates  the  CFM  sigmoid  output  */ 

/♦  for  each  of  the  incorrect  nodes  by  ♦/ 

/♦  2(n)  =  l/[l+exp(-By(c)+By(n)+20ta)]  ♦/ 

/♦  */ 

/*  Functions  Called:  Hone  ♦/ 

/*  Variables  Passed  In:  Hcde.record  -  Structure  array  ♦/ 

/♦  nodes.in.layer  ~  Integer  array  ♦/ 

/♦  starting.node.in.layer  -  Integer  array  ♦/ 

/♦  current .layer  -  Integer  ♦/ 

/■*  correct.node  -  Integer  ♦/ 

/♦  zn  -  Float  array  */ 

/*  CFM.beta  -  Float  ♦/ 

/♦  CFM.zeta  -  Float  ♦/ 

/*  ♦/ 

/*  Variables  Returned:  zn  -  Float  array  ♦/ 

/♦  Date:  10  Hov  90  Revision:  1.0  ♦/ 

/iti*t***********ti¥nn¥**t*t*************************tt**t**************/ 


void  calculate.zn  (struct  Bode.data  ♦node.record [] , 
iiit  nodes.in.layer  [] , 
int  starting.node.in.layer n > 
int  current .layer, 
int  correct.node, 
float  2n[], 
float  beta. 
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float  zeta) 


int  X,;  current^node; 
double  buffer  =  0; 

for  (x  =  0;  X  <  nodes.in.layer [current .layer] ;  x++) 

{ 

current  .node  =  s  t  art  ing.node.in.layer  [current. lay  er]+x; 
buffer  =  zeta  -  beta 

♦  (node.record [correct. node] ->output 
-  node.record  [current  .node]  ->output ) ; 
zn[x]  =  l/(l+exp (buffer)); 

} 


/♦  Function  Hame:  CFM.last .layer  lumber:  8.24  ♦/ 
/*  Description:  This  function  updates  the  weights  of  the  last  */ 
/*  layer  of  a  sigmoidal  network  by  */ 
/♦  w(MH)+  =  w(MI)-anB/(H-l)z(H)[l-2(B)]y(I)[l-y(H)]y(M)  ♦/ 
/♦  «(!)+=  «(H)-anB/(I~l)z(H)[l-2(lI)]y(I)[l-y(H)]  */ 
/♦  w(MC)+  =  w(MC)-anB/(H~l)sum{z(H)[l-z(H)]}y(C)[l-y(C)]y(M)  ♦/ 
/♦  C(C)+  =  «(C)-anB/(H-l)sum{2(H)[l-2(H)]>y(C) [l-y(C)]  ♦/ 
/♦  ♦/ 
/*  Functions  Called:  Vone  */ 
/♦  Variables  Passed  In:  lode.record  -  Structure  array  */ 
/♦  nodes.in.layer  -  Integer  array  ♦/ 
/♦  start ing.node.in.layer  -  Integer  array  ♦/ 
/♦  last.layer  -  Integer  ♦/ 
/♦  zn  -  Float  array  ♦/ 
/*  correct.node  ~  Integer  ♦/ 
/*  CFM.eta  -  Float  */ 
/♦  ♦wght  -  Float  array  */ 
/♦  CFM.moiaentum  -  Float  */ 
/*  */ 
/*  Variables  Returned:  Node.record  -  Structure  Array  ♦/ 
/♦  Date:  10  Hov  90  Revision:  1.0  ♦/ 


/t**’¥****t****i¥*^ti*************************tt‘*************************/ 

void  CFM.last. layer  (struct  Node.data  ♦node.record[]  , 
int  nodes.in.layer  [] , 
int  starting.node.in.layer[] , 
int  last.layer I 
float  2n[] , 
int  correct.node, 
float  eta, 
float  ^wght[], 
float  alpha) 


float  buffer.2,  buffer.l,  sum.zn,  old.wght; 
int  X,  y,  n,  current  .node,  provious.layer.node; 
for  (x  =  0;  X  <  nodes.in.layer[last.layer] ;  x++) 

current  .node  =  starting.node.in.layer[last.layer]+x; 
buffer.l  =  ( 1 -node.r  ecord  [current. node] ->  out  put) 

♦  node.record [current .node] ->output ; 
if  (current.node  !=  correct.node) 

{ 

for  (y  =  0;  y  <  nodes.in.layer [last.layer-1] ;  y++) 

{ 

previous.layer.node  =  start ing.node.in.layer [last .layer-1]  +  y; 
buffer.2  =  node.record  [previous.layer.node]->output; 
old.wght  =  node.record  [current.node] ->weight [previous.layer.node] 
node.record [current.node] ->weight [previous.layer.node]  += 
-Gta*(l-2n[x])*zn[x] 
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♦  buffer«2  ♦  buffer.l 
+  alpha 

♦  (node^record [current «node] ->Height [previous^layer^node] 

-  ♦((ught [current .node]) +previous^layer«node) ) ; 

♦((wght [current .node] )+previou8.1ayer.node)  =  old.wght; 

> 

old.vght  =  node.record[current.node]->sagma[current.node] ; 
node.record[current.ncde]->sigma [current .node]  += 

-eta*(l-zn[x])*2n[x]»  buffer.l 

+  eJLpha  ♦  (node.record[current.node]->sigma [current .node] 
-  ♦ ( (vght [current.node] ) +curr ent.no de) ) ; 

♦((wght [current .node] )+current. node)  =  old.wght; 

} 

else 

{ 

sum.zn  =  0; 

for  (n  =  0;  n  <  nodes.in.layer [last. lay er] ;  n++) 

if  (n  !=  correct.node-starting.node.in.layer [last .layer]) 
sum.zn  =  sum.zn  +  2n[n]*  (l-zn[n]); 

for  (y  =  0;  y  <  nodes.in.layer [last. lay er-1] ;  y++) 

{ 

previous.layer.node  =  starting.node.in.layer[last.layer-l]+y; 
buffer.2  =  node.record[previous.layer.node]“>output; 
old.wght  =  node.record [current .node]->weight [previous.layer.node] ; 
node.r ecord [curr ent. node] ~> weight [previous.layer.node]  += 
eta  *  buffer.l  ♦  buffer.2  ♦  sum.zn 
+  alpha 

♦  (nod e.re c ord [curr ent. node] -> weight [previous.layer.node] 

-  ♦ ( (wght [current.node] )  4prev ious.layer.no de) ) ; 

♦  ((wght [current .node]) +previou8.1ayer .node)  “  old.wght; 

} 

old.wght  =  nod  e.re  cord  [current . node] ->  sigma  [curr  ent. node]  ; 
node.record [current.node] ->sigma [current .node]  += 
eta  ♦  buffer.l  ♦  sum.zn 
+  alpha 

*  (node.record [current.node] ->sigma [current .node] 

-  ♦  (  (wght  [current  .node]  )+curr  ent  .node)  ) ; 

♦ ( (wght [current .node] )+current.node)  =  old.wght ; 

} 

} 

} 


/♦  Function  Same:  CFll.mid.lay or  lumber:  8.25  ♦/ 
/*  Description:  This  function  updates  the  second  hidden  layer  */ 
/♦  parameter  of  a  sigmoidal  network  by  ♦/ 
/♦  w(LM)+ =  w(Lll)-anB/(I-l)sum{z(n)[l-z(n)]}y(c)  [l-y(c)]w(Mc)  ♦/ 
/♦  -  y(n)[l-y(n)]w(Mn)y(M)[l-y(M)]y(L)  */ 
I*  e(M)+  =  Q(M)-anB/(I-l)sum{z(n)[l-2(n)]}y(c)[l~y(c)]w(Mc)  ♦/ 
/♦  -  y(n) [l-y(n)]w(Mn)y(M)[l-y(M)]  ♦/ 
/♦  ♦/ 
/♦  Functions  Called:  lone  ♦/ 
/♦  Variables  Passed  In:  Bode.record  -  Structure  array  ♦/ 
/♦  nodes.in.layer  -  Integer  array  ♦/ 
/♦  starting.node.in.layer  -  Integer  array  ♦/ 
/♦  current  .layer  -  Integer  ♦/ 
/*  zn  “  Float  array  ♦/ 
/*  correct.node  -  Integer  */ 
/♦  CFM.eta  -  Float  ♦/ 
/♦  ♦wght  -  Float  array  ♦/ 
/♦  CFM.momcntum  -  Float  ♦/ 
/*  ♦/ 
/♦  Variables  Returned:  Bode.record  -  Structure  Array  ♦/ 
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/♦  Date:  10  lov  90  Revision:  1.0  ♦/ 


void  CFM.mid_layer (struct  lode.data  ♦node^recordQ , 
int  nodes_in«layerD , 
int  starting.node.in.layern » 
int  currcnt^layer , 
float  znD , 
int  correct^node, 
float  eta, 
float  ♦Bght[], 
float  alpha) 


{ 

int  X,  y,  z,  n; 

int  current  .node,  previous.layer.node ,  last.layer.node ; 
float  buffer.l,  buffer.2,  buffer .3,  buffer.4,  old.wght; 
for  (x  =  0;  X  <  nodes.in.layer [current. lay er] ;  x++) 

{ 

current .node  =  start ing.node. in.layer[current.l ay erj+x; 
buffer.l  =  node.record [current .node] ->output 

♦  (l'-node.record[current_node]“>output) ; 
buffer.3  =  0; 

for  (z  =  0;  z  <  nodcs.in.layer [current .laycr+1] ;  z++) 

{ 

last.layer.node  =  starting.node.in.layer [current _layer+l]  +  z; 
if  (last .layer .node  1=  correct. node) 
buffer.3  +=  zn[z]*(l~zn[z]) 

♦ (1-node.rocord [last.laycr.node] ->output ) 

♦  node.record [last_layer.node] ->output 

♦  node.record [last.layer.node]->weight [current .node]  ; 

else 

{ 

buffer.d  =  0; 

for  (n  =  0;  n  <  nodes.in.layer[currcnt.layer+l] ;  n++) 

if  (n  !=  correct. node-st art ing.node.in.layer [current. lay er+1]) 
buffer.4  =  buffer.4  +  zn[n]*(l-zn[n]) ; 
buffer.4  =  buffer.4  ♦  node_record[correct_node]->output 

♦  (1  -node.record [correct. node] ->output) 

♦  (node.record [correct. node] ->weight [current .node] ) ; 

> 

} 

for  (y  =  0;  y  <  nodes. in.lay er[current. lay e r-1] ;  y++) 

previous.layer.node  =  start ing.node.in.layer [current .lay er-l]+y; 
buffer.2  =  node.record[previous.layer.node]->output; 
old.wght  =  node.record[current.node]->weight[previous.layer.node] ; 
node.record [current . node] ->wcight [previous.layer.node]  *= 
eta  ♦  (buffer.4-buffer_3) 

♦  buffer.l  ♦  buffer.2 
+  alpha 

♦  (node.record [current .node] -> weight [previou8.layer.node] 

“  ♦ ( (wght [curr ent .node] )+pre vious.layer. node)) ; 

♦  ( (wght [current .node] )+previous.layer.aod0)  =  old.wght ; 

} 

old.wght  =  node.record [current .node] ->sigma [curr ent. node] ; 
node.record [curr€nt.nodo]->sigma [curr ent .node]  += 
eta  ♦  (buffcr.4-buffer.3) 

♦  buffer.l 
+  alpha 

♦  (node.record [current .node] ->8igma [current .node] 

-  ♦ ( (wght [curr ent .node] )+current.nodc) ) ; 

♦  ( (wght [current  jiodc] )+curr ent .node)  =  old.wght ; 


} 
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} 


/*  Function  Same:  find^second^highost^node  Humber:  8.26  */ 
/*  Description:  This  function  returns  the  incorrect  node  with  the  ♦/ 
/♦  highest  output  value  ♦/ 
I*  Functions  Called: Hone  ♦/ 
/♦  Variables  Passed  In:  Hode.record  -  Structure  array  ♦/ 
/*  nodes.in.layer  -  Integer  array  ♦/ 
/♦  start ing.node«in.layer  ~  Integer  array  ♦/ 
/*  last^layer  -  Integer  ♦/ 
/♦  winner^nodo  -  Integer  */ 
/♦  ♦next«highest«node  -  Integer  pointer  ♦/ 
/♦  ♦/ 
/♦  Variables  Returned:  >»next .highest .node  -  Integer  pointer  ♦/ 
/♦  Date:  10  Hov  90  Revision:  1.0  ♦/ 


void  f ind.se cond.highest. node  (struct  Hode.data  *node.record[] , 

int  nodes. in.layer[] , 

int  start ing.node.in.layer C3 , 

int  last.layer, 

int  winner.node, 

int  *next.highest.node) 

{ 

int  X,  current.node ; 
float  outmax  -  0; 

for  (x  =  0;  X  <  nodes.in.layer [last.layer] ;  x++) 

{ 

current.node  =  starting.node.in.layer[last_layer]+x; 
if  (current.node  !=  winner.node) 

{ 

if  (node. re  cord  [current.node] ->  out  put  >  outmax) 

{ 

outmax  =  no de.r e cord [current .node] -> out put; 
♦next.highest.node  =  current.node; 

} 

} 

} 

} 


/♦  Function  Hame:  CFM.first.layer  Humber:  8.27  ♦/ 
/«  Description:  This  function  updates  the  first  hidden  layer  of  a  *1 
/♦  three  layer  sigmoidal  network  by  ♦/ 
/♦  w(KL)+  =  w(KL)-anB/(I-l)sum{z(n)[l-z(n)]}y(c)[l-y(c)]sum{w(mc}  ♦/ 
/♦  -y(n)  [l-y(n)]sum{w(mn)y(m)ri-y(m)]w(Lia)y(L)Cl-y(L)y(K)  ♦/ 
/♦  fi(KL)+  =  ®(KL)-anB/(H-l)sum{z(n)[l-z(n)]}y(c)[l-y(c)]sum{w(mc}  ♦/ 
/♦  “y(n)  [l~y(n)3sum{w(mn)y(m)[l“y(m)]w(Lm)y(L)[l-y(L)  ♦/ 
/♦  »/ 
/*  Functions  Called:  Hone  */ 
/♦  Variables  Passed  In;  Hode.record  -  Structure  array  ♦/ 
/♦  nodes.in.layer  -  Integer  array  ♦/ 
/♦  start ing_node.in.layer  -  Integer  array  ♦/ 
/♦  current. layer  -  Integer  ♦/ 
/*  zn  -  Float  array  ♦/ 
/♦  correct. node  -  Integer  ♦/ 
/♦  CFM.eta  -  Float  ♦/ 
/♦  ♦wght  “  Float  array  */ 
/♦  CFH.momentum  -  Float  */ 
/♦  total.nodcs  -  Integer  */ 
/*  */ 
/♦  Variables  Returned:  Hode.record  -  Structure  Array  ♦/ 
/♦  Date:  10  Hov  90  Revision:  1.0  ♦/ 
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void  CFM.first.layer  (struct  lodo.data  ♦node.recordQ , 
int  nodes«iii.lay«r[3, 
int  St  art ing^node. in.lay or [] , 
int  current. layer, 
float  znC] , 
int  correct.node , 
float  eta, 
float  *ught [] , 
float  alpha, 
int  total.nodes) 


int  X,  y,  z; 

float  buffer.l,  buffer.2,  old.wght; 

int  first.lyr.node,  last. lyr .node,  nid.lyr.node ,  input .lyr. node; 
float  output [TOTAL.HODES] ; 

for  (x  =  0;  X  <  total.nodes;  x++) 

output [x]  =  (l~node.recordCx]->output)^node.recordCx]->output; 

for  (x  =  0;  X  <  nodes.in.layer [current .lay er] ;  x++) 

{ 

buffer.2  =  0; 

f irst.lyr.node  =  start ing.node.in.laycr [current .layer]  +  x; 
for  (y  =  0;  y  <  nodes.in.layer [current.layer+23 ;  y++) 

{ 

last.lyr.node  =  starting.node.in.layer [current .layer+2]  +  y; 
if  (last.lyr.node  !=  correct.node) 

buffer.l  =  0; 

for  (z  =  0;  z  <  nodes.in.layer [current .layer+1] ;  z++) 

nid.lyr.node  =  st arting.nodo.in.layer [current .layer+l3+z; 
buffer.l  =  buffer.l 

+  (output [correct.node] ♦node.record [correct.node] -> weight [nid.lyr.node] 

-  output [last.lyr.node] vnode.record [last.lyr.node] ->weight [nid.lyr.node] ) 
*  output [mid.lyr jiode] ♦node.record[nid.lyr.node]->weight[f irst.lyr.node] ; 

> 

buffer.2  =  buffer.2  +  zn[y]*(l-zn[y])*  buffer.l; 

} 

} 

for  (z  =  0;  z  <  nodes.in.layer [currcnt.layer-1] ;  z++) 

{ 

input  .lyr. node  =  start  ingjiode.iii.layer  [current  .layer-1]  +  z; 
old.wght  =  * ( (wght [f ir st.lyr.no de] )+input .lyr .node) ; 

♦ ( (wght [f irst.lyr.node] )+ input .lyr. node)  = 

node.record[first.lyr.node]-> weight [input .lyr.node] ; 

node.record[first.lyr.node]->weight [input .lyr  jiode]  += 
eta  ♦  buffer.2  ♦  output [first. lyr.node] 

*  node.record[input.lyr  jiode] ->output 
+  alpha 

♦  (node.record [first .lyr.node] ->weight [input .lyr .node] 

-  old.wght) ; 


> 

/ 

old.wght  =  *( (wght [first.lyr. node] )+f irst. lyr.node) ; 

♦ ( (wght [fir St. lyr .node] )+first. lyr. node)  = 

node.record [first.lyr.node]->sigma [first. lyr .node] ; 
node.record [f irst.lyr.node] ->sigma [f irst.lyr.node]  += 

eta  •  buffcr.2  ♦  output [first.lyr. node] 

+  alpha 
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> 


♦  (iiode«record[first.lyr_node3*>sigma[first.lyr«node] 
“  old^wght) ; 

> 


/*  End  Functions  Called  by  CFM  Remaining  Lyrs  */ 

/^t******t*tlt*^*t^r¥*^t****4^*t***^t*************4**t***/ 

/44444*444444*4444444444*4444*44*44**444***4*4*4*******44*44*444*444/ 

/♦  Function  lame:  find.nearest^neighbor  Humber :8. 28  ♦/ 

/♦  Description:  This  function  finds  the  euclidean  distance  between  */ 
!*  nodes  in  the  seime  layer  ♦/ 

/♦  d(ij)  =  sqrt{sumCw(i)-w(j)]^2}  ♦/ 

/♦  Functions  Called:  Hone  ♦/ 

/♦  Variables  Passed  In;  Hode.record  -  Structure  array  */ 

/♦  nodes«in_layer  -  Integer  array  ♦/ 

/♦  starting^node.in^layer  -  Integer  array  ♦/ 

/♦  current^node  -  Integer  */ 

/♦  next.node  -  Integer  ♦/ 

/♦  current^layer  ~  Integer  ♦/ 

/♦  ♦distancG^between  -  Float  array  pointer  ♦/ 

/*  */ 

/♦  Variables  Returned:  ♦distance^between  -  pointer  ♦/ 

/♦-Date:  10  Hov  90  Revision:  1.0  ♦/ 


f 444*44444*44***4*4**444*4*44*444*4444444444*44*44*4*4*4*4*44*4444*4/ 


void  f ind^nearest_neighbor(struct  data  ♦data.recordC] , 

struct  Hode.data  ♦node.record[] , 
int  record, 
int  nodes^in.layerQ , 
int  starting.node.in^layerC] , 
int  current .layer, 
int  ♦nearest.node) 


{ 


int  X,  y,  current .node,  previous.layer.node; 
float  buffer,  distance,  nearest .distance; 
double  exponent  =  2; 
calculate.layer.O.output (data.r ecord , 
node.record, 
nodes.in.layer , 
record) ; 

nearest.distance  =  1000; 

for  (x  =  0;  X  <  nodes.in.layer [current. layer] ;  x++) 

current .node  =  start ing.node.in.layer [current .layer]  +  x; 
distance  =  0; 

for  (y  =  0;  y  <  nodes.in.layer  [current. lay  er-1] ;  y++) 

previous.layer.node  =  start ing.node.in_layer[current.layor-l]+y; 
buffer  =  node.record [current . node] ->weight[previous.layer.node] 

-  Mode.rccord [previous.layer.node] ->output ; 
distance  =  distance  +  powCbuffor, exponent) ; 

} 

if  (distance  <  nearest.dist2Lnce) 

-C 

nearest.distance  =  distance; 

♦nearest.node  =  current.node; 

} 

> 
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} 


G.9  NETMATH 


/*  Module  Barne:  BETMATfl.C  lumber:  9.0  ♦/ 
/*  Description:  This  nodule  contains  the  basic  mathematical  */ 
/♦  used  to  train  the  networks  ♦/ 
/*  ♦/ 
/♦  Modules  Called:  lone  »/ 
/*  Functions  Contained:  */ 
/♦  9.1  make.identity^matrix  9.2  determine joatrix.transpose  */ 
/♦  9.3  invert. a.aatrix  9.4  update.average  ♦/ 
/♦  9.5  update.sigma  9.6  chlculate.percentage  ♦/ 
/♦  ♦/ 
/*  Date:  11  Sov  90  Revision:  1.0  ♦/ 


/tt^^f^t*************************************************************/ 


t include  "netvrble .h" 
t include  '*netfnctn.h" 


/♦  Function  lame:  make.identity. matrix  Humber:  9.1  ♦/ 
/*  Description:  This  function  returns  a  matrix  whose  elements  are  */ 
/*  are  1  on  the  diagonal  and  0  elsewhere  */ 
/*  */ 
/*  Functions  Called:  Hone  */ 
/♦  Variables  Passed  In:*H  -  Float  array  pointer  ♦/ 
/♦  nodes.2  -  Integer  ♦/ 
/*  ♦/ 
/♦  Variables  Returned:  ♦!  -  Float  array  pointer  ♦/ 
/♦  Date:  11  Ko^  90  Revision:  1.0  ♦/ 


void  mako.idontityjaatriy  (float  ♦HC3, 

int  nodes.2) 

{ 

int  row,  column; 

for  (row  =  0;  row  <  nodes„2;  xOw++) 
for  (column  =  C,  column  <  nodes_2; 
if  (row  ==  Column) 
♦((lCrow3)+colunn)  =  1; 
else 

♦  ((HCrow3)+coluinn)  =  0; 


column++) 


/*  Function  HalAe:  determiliB.matrix.transpose  Humber:  9.2  */ 

/♦  Description:  This  sanction  returns  Vie  tra’^-xpof  j  of  a  square  ♦/ 
/*  matrix  by  ♦/ 

/*  KTCx3Cy]  =  MCy3Cx3  */ 

/♦  »/ 

/♦  Functions  Called;  Hone  ♦/ 

/*  Variables  Passeu  tji:  *MT  -  Float  «^rray  pointer  ♦/ 

/♦  ♦M  -  Float  arx;ay  pointer  */ 

/♦  total.rbfs  ♦/ 

/*  ♦/ 

/♦  Variables  Returned:  '*'MT  -  Float  array  pointer  ♦/ 

/v  Dato:  11  Hov  90  Rev^’^ion:  1.0  ♦/ 
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void  detenaine^matrix^transpose  (float  *MTC3, 

float  ♦MC3, 
int  total.rbfs) 

{ 

int  rov,  coltimn; 

for  (row  =  0;  row  <  total^rbfs;  row  ++) 

for  (column  =  0;  column  <  total.rbfs;  column++) 
♦  ((MTCrowD+column)  =  ♦  ((H  [column]  ) +row) ; 
for  (row  =  0;  row  <  total.rbfs;  row++) 

for  (column  =  0;  column  <  total.rbfs;  column++) 
♦((M[row3)+column)  =  ♦((MT[row3)+column) ; 

} 


/*  Function  lame;  invort^ajnatrix  Vumber:  9.3  ♦/ 
/*  Description:  This  function  returns  inverts  a  square  matrix  */ 
/♦  via  Gaussian  elmination  ♦/ 
/*  */ 
/♦  Functions  Called:  lone  ♦/ 
/♦  Variables  Passed  In:  ♦MC3  “  Float  array  pointer  ♦/ 
/♦  ♦H[3  ~  Float  array  pointer  ♦/ 
/♦  nodes.2  -  Integer  ♦/ 
/♦  ♦/ 
/*  Variables  Returned:  ♦HC3  -  Float  array  pointer  ♦/ 
/*  Date:  11  lov  90  Revision:  1.0  */ 


void  invert^a_matrix  (float  ♦MG, 
float  ♦I[3, 
int  nodes.2) 

{ 

float  beta  =  0; 

float  alpha  =  0; 

int  XX,  yy,  row,  colxuon; 

for  (row  =  0;  row  <  nodes_2;  row++) 

{ 

beta  =  *((M[row3)+row) ; 

for  (column  =  0;  column  <  nodes_2;  column++) 

{ 

♦((M[row3)+column)  =  ♦((M[row3)+column)/beta: 
♦((N[row3)+column)  =  *((N[row3)+column)/beta; 

} 

for  (xx  =0;  xx  <  nodes.2;  xx++) 
if  (xx  !=  row) 

{ 

alpha  =  ♦((M[xx3)+row) ; 

for  (yy  =  0;  yy  <  nodes.2;  yy++) 

{ 

♦((M[xx3)+yy)  =  -alpha  ♦  *((MCrow3)+yy)  +  ♦((MCxx3)+yy) ; 
♦  ((irCxx3)+yy)  =  -alpha  ♦  ♦((HCrow3)+yy)  +  ♦((H[xx3)+yy) ; 
} 

} 

} 

} 


/*  Function  Hame:  update.average  Bumber:  9.4  */ 
/*  Descr7ption:  This  function  maintains  a  running  average  */ 
/♦  avg(+)  =  avg(-)+[x-avg(-)3/[H(-)+l3  ♦/ 
/♦  ♦/ 
/♦  Functions  Called:  Hone  */ 
/*  Variables  Passed  In:  current. average  -  Float  ♦/ 
/♦  elements  -  Integer  ♦/ 
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/♦  X  -  Float  ♦/ 
/♦  •next^average  -  Float  pointer  ►/ 
/♦  ♦/, 
/♦  Variables  Returned:  ♦next .average  -  Float  pointer  ♦/ 
/♦  Date:  11  lov  90  ReTision:  1.0  ♦/ 


/#««#***♦♦♦♦♦♦♦*♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦*♦♦♦♦♦♦♦♦♦*♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦/ 

void  update^average  (float  current.avorage, 
int  elements, 
float  X, 

float  ♦next^average) 


{ 

float  y; 

♦next^average  “  current«average  +  (1/ (float) elements) ♦(x-current ^average); 

} 


/♦  Function  lame:  update.sigma  lumber: 9. 5  ♦/ 
/♦  Description:  This  function  maintains  a  running  sigma  by  ♦/ 
/*  sigma(*i-)  =  sqrt{sigma(-)^2+(l/M)  [1**(1/I)]  Cx-avg(-)]"2-sigma(-)"2}  ♦/ 
/♦  ♦/ 
/*  Functions  Called:  lone  ♦/ 
/♦  Variables  Passed  In:  current.sigma  -  Float  ♦/ 
/*  elements  -  Integer  ♦/ 
/♦  X  -  Integer  ♦/ 
/•  current.average  -  Float  ♦/ 
/*  ♦next.sigma  -  Float  pointer  ♦/ 
/*  */ 
/♦  Variables  Returned:  ♦next^sigma  -  Float  pointer  ♦/ 
/♦  Date:  11  lov  90  Revision:  1.0  ♦/ 


void  updaters igma  (float  current. sigma, 
int  elements, 
float  X, 

float  current.average, 
float  ♦next.sigma) 

{ 

♦next.sigma  =  pow ((double) (current .sigma) ,2)+(l/ (float )elements) 

♦  ((l~l/(float)elements)  ♦  pow ( (double) (x-current.average) ,2) 
-  pov( (double) (current .sigma) ,2) ) ; 

♦next. sigma  =  po«( (double) (♦next.sigma) , .5) ; 

> 


/*  Function  Kamo:  calculate.percentage  Kumber:  9.6  ♦/ 
/♦  Descriptior:  This  function  returns  the  percentage  value  ♦/ 
/♦  ♦/ 
/♦  Functions  Called:  Hone  ♦/ 
/♦  Variables  Passed  In:  numerator  -  Float  ♦/ 
/♦  denominator  -  Float  ♦/ 
/♦  ♦per.cent  -  Float  pointer  ♦/ 
/*  ♦/ 
/*  Variables  Returned:  ♦per.cent  -  Float  pointer  ♦/ 
/♦  Date:  11  Kov  90  Revision:  1.0  ♦/ 


/^0*^1^Jt^^*************i******^***^**t*******^*****************i.*******/ 

void  calculate.percentage (float  numerator, 

float  denominator, 
float  ♦per.cent) 

{ 

♦per.cent  =  100  ♦  numeral or/denominat or; 

} 
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GJO  NETVRBLE 


tinclxjde  <stdio.h> 
t include  <m&th.h> 


/«  Isose:  VETVRBLE  lumber:  10*0  */ 

/♦  Description:  This  module  contains  the  data  structures  ♦/ 
/*  for  the  nodes  and  the  data  along  with  ♦/ 

/*  the  marimum  values  for  the  global  */ 

/*  variables  */ 

/♦  ♦/ 
/♦  Modules  Called:  Hone  ♦/ 

/♦  Functions  Contained:  None  */ 

/♦  Variables  Passed  In:  Hone  ♦/ 

/♦  Variables  Returned:  Hone  */ 

/♦  Date:  10  Hov  90  Revision:  1.0  ♦/ 

/i^0t**********i^****t^i^*******f¥**t****************4t*’$*****^t**/ 

tdefine  DIMEHSIOH  100  /*  Length  of  Feature  Vector  */ 

tdefine  TRAIH.SET  200  /♦  Humber  of  Training  Patterns  ♦/ 

tdefine  TEST.SET  200  /♦  Humber  of  Test  Patterns  ♦/ 

tdefine  CLASSES  20  /*  Humber  of  Classes  */ 

tdefine  TOTAL.HODES  250  /*  Humber  of  Nodes  ♦/ 

struct  data 

{ 

float  vector [DIMEHSIOH]; 
int  class; 
int  number; 

}; 

Struct  Node.data 

{ 

float  weight [TOTAL.HODES]; 
float  sigma [TOTAL.HODES]; 
int  class; 

int  transfor.functioii; 

float  output; 

int  connect [TOTAL.HODES] ; 
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